Oct 30 20:25:45.303138 ip-10-0-139-149 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 20:25:45.727597 ip-10-0-139-149 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:25:45.727597 ip-10-0-139-149 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 20:25:45.727597 ip-10-0-139-149 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:25:45.727597 ip-10-0-139-149 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:25:45.734481 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734387    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:25:45.734481 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734481    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:25:45.734552 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734487    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:25:45.734552 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734492    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:25:45.734552 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734496    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:25:45.734552 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734499    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:25:45.734552 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734503    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:25:45.734552 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734506    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:25:45.734552 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734508    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:25:45.734552 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734511    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:25:45.734552 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734513    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:25:45.734552 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734516    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:25:45.734552 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734519    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:25:45.734552 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734521    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:25:45.734552 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734524    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:25:45.734552 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734534    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:25:45.734552 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734537    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:25:45.734552 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734539    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:25:45.734552 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734542    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:25:45.734552 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734545    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:25:45.734552 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734547    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:25:45.735000 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734550    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:25:45.735000 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734552    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:25:45.735000 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734555    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:25:45.735000 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734558    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:25:45.735000 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734560    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:25:45.735000 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734563    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:25:45.735000 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734566    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:25:45.735000 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734572    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:25:45.735000 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734575    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:25:45.735000 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734578    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:25:45.735000 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734580    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:25:45.735000 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734582    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:25:45.735000 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734585    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:25:45.735000 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734588    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:25:45.735000 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734590    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:25:45.735000 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734592    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:25:45.735000 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734595    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:25:45.735000 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734597    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:25:45.735000 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734600    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:25:45.736069 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734602    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:25:45.736069 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734605    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:25:45.736069 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734607    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:25:45.736069 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734610    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:25:45.736069 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734612    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:25:45.736069 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734615    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:25:45.736069 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734618    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:25:45.736069 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734621    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:25:45.736069 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734623    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:25:45.736069 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734627    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:25:45.736069 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734632    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:25:45.736069 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734634    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:25:45.736069 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734637    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:25:45.736069 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734640    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:25:45.736069 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734643    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:25:45.736069 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734645    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:25:45.736069 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734648    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:25:45.736069 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734650    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:25:45.736069 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734653    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:25:45.736069 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734655    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:25:45.736534 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734658    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:25:45.736534 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734661    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:25:45.736534 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734664    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:25:45.736534 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734666    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:25:45.736534 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734669    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:25:45.736534 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734671    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:25:45.736534 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734674    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:25:45.736534 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734676    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:25:45.736534 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734679    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:25:45.736534 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734681    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:25:45.736534 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734684    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:25:45.736534 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734686    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:25:45.736534 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734689    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:25:45.736534 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734692    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:25:45.736534 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734694    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:25:45.736534 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734698    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:25:45.736534 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734700    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:25:45.736534 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734703    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:25:45.736534 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734706    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:25:45.737008 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734709    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:25:45.737008 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734711    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:25:45.737008 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734713    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:25:45.737008 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734716    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:25:45.737008 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734719    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:25:45.737008 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734721    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:25:45.737008 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734724    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:25:45.737008 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734726    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:25:45.737008 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734728    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:25:45.737008 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734731    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:25:45.737008 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734733    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:25:45.737008 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734735    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:25:45.737008 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734738    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:25:45.737008 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734741    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:25:45.737008 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734743    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:25:45.737008 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734746    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:25:45.737008 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734749    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:25:45.737008 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734751    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:25:45.737008 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734754    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:25:45.737008 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734756    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:25:45.737470 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734760    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:25:45.737470 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734762    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:25:45.737470 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734765    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:25:45.737470 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734767    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:25:45.737470 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734770    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:25:45.737470 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734772    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:25:45.737470 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.734775    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:25:45.737470 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736301    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:25:45.737470 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736309    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:25:45.737470 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736313    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:25:45.737470 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736317    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:25:45.737470 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736320    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:25:45.737470 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736323    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:25:45.737470 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736326    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:25:45.737470 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736328    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:25:45.737470 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736332    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:25:45.737470 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736334    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:25:45.737470 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736336    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:25:45.737470 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736339    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:25:45.737948 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736341    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:25:45.737948 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736344    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:25:45.737948 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736347    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:25:45.737948 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736349    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:25:45.737948 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736352    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:25:45.737948 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736355    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:25:45.737948 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736357    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:25:45.737948 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736359    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:25:45.737948 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736362    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:25:45.737948 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736365    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:25:45.737948 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736367    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:25:45.737948 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736370    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:25:45.737948 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736372    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:25:45.737948 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736376    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:25:45.737948 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736378    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:25:45.737948 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736381    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:25:45.737948 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736384    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:25:45.737948 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736386    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:25:45.737948 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736388    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:25:45.737948 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736391    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:25:45.738429 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736395    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:25:45.738429 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736398    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:25:45.738429 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736401    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:25:45.738429 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736405    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:25:45.738429 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736407    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:25:45.738429 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736409    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:25:45.738429 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736412    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:25:45.738429 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736415    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:25:45.738429 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736417    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:25:45.738429 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736419    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:25:45.738429 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736422    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:25:45.738429 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736425    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:25:45.738429 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736428    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:25:45.738429 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736430    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:25:45.738429 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736433    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:25:45.738429 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736435    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:25:45.738429 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736438    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:25:45.738429 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736441    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:25:45.738429 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736444    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:25:45.738887 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736446    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:25:45.738887 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736448    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:25:45.738887 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736451    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:25:45.738887 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736454    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:25:45.738887 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736457    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:25:45.738887 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736459    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:25:45.738887 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736462    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:25:45.738887 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736464    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:25:45.738887 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736467    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:25:45.738887 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736469    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:25:45.738887 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736472    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:25:45.738887 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736474    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:25:45.738887 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736476    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:25:45.738887 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736479    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:25:45.738887 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736481    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:25:45.738887 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736483    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:25:45.738887 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736487    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:25:45.738887 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736489    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:25:45.738887 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736492    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:25:45.738887 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736494    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:25:45.739347 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736497    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:25:45.739347 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736499    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:25:45.739347 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736502    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:25:45.739347 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736504    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:25:45.739347 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736507    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:25:45.739347 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736509    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:25:45.739347 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736511    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:25:45.739347 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736514    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:25:45.739347 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736516    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:25:45.739347 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736519    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:25:45.739347 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736521    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:25:45.739347 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736523    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:25:45.739347 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736526    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:25:45.739347 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736528    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:25:45.739347 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736531    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:25:45.739347 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736534    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:25:45.739347 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736537    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:25:45.739347 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736540    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:25:45.739347 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736543    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:25:45.740052 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736546    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:25:45.740052 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736548    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:25:45.740052 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736551    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:25:45.740052 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736553    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:25:45.740052 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736556    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:25:45.740052 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736558    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:25:45.740052 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736561    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:25:45.740052 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736563    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:25:45.740052 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736566    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:25:45.740052 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736568    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:25:45.740052 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736573    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:25:45.740052 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736577    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:25:45.740052 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736580    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:25:45.740052 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736583    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:25:45.740052 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736585    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:25:45.740052 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.736588    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:25:45.740052 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737091    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:25:45.740052 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737098    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:25:45.740052 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737101    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:25:45.740728 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737104    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:25:45.740728 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737108    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:25:45.740728 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737111    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:25:45.740728 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737114    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:25:45.740728 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737117    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:25:45.740728 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737119    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:25:45.740728 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737122    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:25:45.740728 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737125    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:25:45.740728 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737128    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:25:45.740728 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737132    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:25:45.740728 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737136    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:25:45.740728 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737139    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:25:45.740728 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737142    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:25:45.740728 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737144    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:25:45.740728 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737147    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:25:45.740728 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737149    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:25:45.740728 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737152    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:25:45.740728 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737154    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:25:45.740728 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737157    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:25:45.741197 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737160    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:25:45.741197 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737162    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:25:45.741197 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737164    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:25:45.741197 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737167    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:25:45.741197 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737170    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:25:45.741197 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737172    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:25:45.741197 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737175    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:25:45.741197 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737177    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:25:45.741197 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737179    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:25:45.741197 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737182    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:25:45.741197 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737184    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:25:45.741197 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737187    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:25:45.741197 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737189    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:25:45.741197 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737191    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:25:45.741197 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737194    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:25:45.741197 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737196    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:25:45.741197 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737198    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:25:45.741197 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737201    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:25:45.741197 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737203    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:25:45.741197 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737205    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:25:45.741686 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737208    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:25:45.741686 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737211    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:25:45.741686 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737213    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:25:45.741686 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737216    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:25:45.741686 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737219    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:25:45.741686 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737221    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:25:45.741686 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737224    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:25:45.741686 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737226    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:25:45.741686 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737229    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:25:45.741686 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737231    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:25:45.741686 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737234    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:25:45.741686 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737237    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:25:45.741686 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737239    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:25:45.741686 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737241    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:25:45.741686 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737244    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:25:45.741686 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737247    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:25:45.741686 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737249    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:25:45.741686 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737251    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:25:45.741686 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737254    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:25:45.742171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737256    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:25:45.742171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737259    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:25:45.742171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737261    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:25:45.742171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737263    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:25:45.742171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737266    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:25:45.742171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737268    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:25:45.742171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737270    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:25:45.742171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737273    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:25:45.742171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737275    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:25:45.742171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737278    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:25:45.742171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737280    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:25:45.742171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737283    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:25:45.742171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737286    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:25:45.742171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737288    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:25:45.742171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737291    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:25:45.742171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737293    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:25:45.742171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737295    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:25:45.742171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737298    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:25:45.742171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737301    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:25:45.742171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737303    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:25:45.742636 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737306    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:25:45.742636 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737308    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:25:45.742636 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737310    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:25:45.742636 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737313    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:25:45.742636 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737315    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:25:45.742636 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737317    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:25:45.742636 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737320    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:25:45.742636 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737322    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:25:45.742636 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737327    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:25:45.742636 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737329    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:25:45.742636 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737332    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:25:45.742636 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737334    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:25:45.742636 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737337    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:25:45.742636 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737340    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:25:45.742636 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737342    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:25:45.742636 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737345    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:25:45.742636 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737347    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:25:45.742636 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737349    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:25:45.742636 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737352    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:25:45.743109 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737355    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:25:45.743109 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737357    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:25:45.743109 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737359    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:25:45.743109 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737362    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:25:45.743109 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737364    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:25:45.743109 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.737369    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:25:45.743109 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738279    2575 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 20:25:45.743109 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738290    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 20:25:45.743109 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738297    2575 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 20:25:45.743109 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738302    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 20:25:45.743109 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738306    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 20:25:45.743109 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738309    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 20:25:45.743109 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738314    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 20:25:45.743109 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738318    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 20:25:45.743109 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738321    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 20:25:45.743109 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738324    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 20:25:45.743109 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738328    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 20:25:45.743109 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738331    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 20:25:45.743109 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738334    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 20:25:45.743109 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738337    2575 flags.go:64] FLAG: --cgroup-root=""
Oct 30 20:25:45.743109 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738340    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 20:25:45.743109 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738345    2575 flags.go:64] FLAG: --client-ca-file=""
Oct 30 20:25:45.743624 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738348    2575 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 20:25:45.743624 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738350    2575 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 20:25:45.743624 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738354    2575 flags.go:64] FLAG: --cluster-domain=""
Oct 30 20:25:45.743624 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738357    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 20:25:45.743624 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738360    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 20:25:45.743624 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738363    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 20:25:45.743624 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738366    2575 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 20:25:45.743624 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738370    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 20:25:45.743624 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738373    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 20:25:45.743624 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738376    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 20:25:45.743624 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738379    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 20:25:45.743624 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738382    2575 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 20:25:45.743624 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738385    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 20:25:45.743624 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738388    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 20:25:45.743624 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738391    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 20:25:45.743624 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738394    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 20:25:45.743624 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738398    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 20:25:45.743624 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738401    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 20:25:45.743624 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738404    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 20:25:45.743624 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738407    2575 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 20:25:45.743624 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738410    2575 flags.go:64] FLAG: --enable-server="true"
Oct 30 20:25:45.743624 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738412    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 20:25:45.743624 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738416    2575 flags.go:64] FLAG: --event-burst="100"
Oct 30 20:25:45.743624 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738420    2575 flags.go:64] FLAG: --event-qps="50"
Oct 30 20:25:45.743624 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738423    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 20:25:45.744232 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738426    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 20:25:45.744232 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738428    2575 flags.go:64] FLAG: --eviction-hard=""
Oct 30 20:25:45.744232 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738433    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 20:25:45.744232 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738436    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 20:25:45.744232 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738439    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 20:25:45.744232 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738442    2575 flags.go:64] FLAG: --eviction-soft=""
Oct 30 20:25:45.744232 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738446    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 20:25:45.744232 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738448    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 20:25:45.744232 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738451    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 20:25:45.744232 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738454    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 20:25:45.744232 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738457    2575 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 20:25:45.744232 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738459    2575 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 20:25:45.744232 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738462    2575 flags.go:64] FLAG: --feature-gates=""
Oct 30 20:25:45.744232 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738466    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 20:25:45.744232 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738469    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 20:25:45.744232 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738472    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 20:25:45.744232 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738475    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 20:25:45.744232 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738478    2575 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 20:25:45.744232 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738481    2575 flags.go:64] FLAG: --help="false"
Oct 30 20:25:45.744232 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738483    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:25:45.744232 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738487    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 20:25:45.744232 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738490    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 20:25:45.744232 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738492    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 20:25:45.744787 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738496    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 20:25:45.744787 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738499    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 20:25:45.744787 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738502    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 20:25:45.744787 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738505    2575 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 20:25:45.744787 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738508    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 20:25:45.744787 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738510    2575 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 20:25:45.744787 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738513    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 20:25:45.744787 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738516    2575 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 20:25:45.744787 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738519    2575 flags.go:64] FLAG: --kube-reserved=""
Oct 30 20:25:45.744787 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738522    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 20:25:45.744787 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738525    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 20:25:45.744787 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738528    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 20:25:45.744787 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738530    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 20:25:45.744787 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738533    2575 flags.go:64] FLAG: --lock-file=""
Oct 30 20:25:45.744787 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738535    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 20:25:45.744787 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738538    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 20:25:45.744787 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738541    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 20:25:45.744787 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738546    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 20:25:45.744787 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738549    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 20:25:45.744787 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738552    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 20:25:45.744787 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738555    2575 flags.go:64] FLAG: --logging-format="text"
Oct 30 20:25:45.744787 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738557    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 20:25:45.744787 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738561    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 20:25:45.744787 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738563    2575 flags.go:64] FLAG: --manifest-url=""
Oct 30 20:25:45.745353 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738566    2575 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 20:25:45.745353 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738570    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 20:25:45.745353 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738573    2575 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 20:25:45.745353 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738577    2575 flags.go:64] FLAG: --max-pods="110"
Oct 30 20:25:45.745353 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738580    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 20:25:45.745353 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738583    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 20:25:45.745353 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738585    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 20:25:45.745353 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738588    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 20:25:45.745353 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738591    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 20:25:45.745353 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738594    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 20:25:45.745353 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738596    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 20:25:45.745353 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738603    2575 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 20:25:45.745353 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738606    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 20:25:45.745353 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738609    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 20:25:45.745353 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738612    2575 flags.go:64] FLAG: --pod-cidr=""
Oct 30 20:25:45.745353 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738615    2575 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 20:25:45.745353 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738618    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 20:25:45.745353 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738621    2575 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 20:25:45.745353 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738624    2575 flags.go:64] FLAG: --port="10250"
Oct 30 20:25:45.745353 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738627    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 20:25:45.745353 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738630    2575 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-04bcadb1c15a1ecbe"
Oct 30 20:25:45.745353 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738633    2575 flags.go:64] FLAG: --qos-reserved=""
Oct 30 20:25:45.745353 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738636    2575 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 20:25:45.745353 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738638    2575 flags.go:64] FLAG: --register-node="true"
Oct 30 20:25:45.745353 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738641    2575 flags.go:64] FLAG: --register-with-taints=""
Oct 30 20:25:45.745353 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738645    2575 flags.go:64] FLAG: --registry-burst="10"
Oct 30 20:25:45.745963 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738647    2575 flags.go:64] FLAG: --registry-qps="5"
Oct 30 20:25:45.745963 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738650    2575 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 20:25:45.745963 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738653    2575 flags.go:64] FLAG: --reserved-memory=""
Oct 30 20:25:45.745963 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738656    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 20:25:45.745963 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738659    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 20:25:45.745963 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738662    2575 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 20:25:45.745963 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738665    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 20:25:45.745963 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738667    2575 flags.go:64] FLAG: --runonce="false"
Oct 30 20:25:45.745963 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738670    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 20:25:45.745963 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738673    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 20:25:45.745963 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738676    2575 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 20:25:45.745963 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738679    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 20:25:45.745963 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738682    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 20:25:45.745963 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738685    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 20:25:45.745963 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738687    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 20:25:45.745963 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738690    2575 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 20:25:45.745963 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738693    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 20:25:45.745963 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738696    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 20:25:45.745963 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738699    2575 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 20:25:45.745963 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738701    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 20:25:45.745963 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738704    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 20:25:45.745963 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738707    2575 flags.go:64] FLAG: --system-cgroups=""
Oct 30 20:25:45.745963 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738710    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 20:25:45.745963 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738715    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 20:25:45.745963 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738719    2575 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 20:25:45.746588 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738721    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 20:25:45.746588 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738725    2575 flags.go:64] FLAG: --tls-min-version=""
Oct 30 20:25:45.746588 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738729    2575 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 20:25:45.746588 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738732    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 20:25:45.746588 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738735    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 20:25:45.746588 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738738    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 20:25:45.746588 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738740    2575 flags.go:64] FLAG: --v="2"
Oct 30 20:25:45.746588 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738748    2575 flags.go:64] FLAG: --version="false"
Oct 30 20:25:45.746588 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738752    2575 flags.go:64] FLAG: --vmodule=""
Oct 30 20:25:45.746588 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738756    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 20:25:45.746588 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.738759    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 20:25:45.746588 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.738907    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:25:45.746588 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.738911    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:25:45.746588 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.738914    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:25:45.746588 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.738916    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:25:45.746588 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.738919    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:25:45.746588 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.738922    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:25:45.746588 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.738924    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:25:45.746588 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.738927    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:25:45.746588 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.738930    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:25:45.746588 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.738932    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:25:45.746588 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.738934    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:25:45.747120 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.738937    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:25:45.747120 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.738939    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:25:45.747120 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.738942    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:25:45.747120 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.738944    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:25:45.747120 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.738947    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:25:45.747120 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.738951    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:25:45.747120 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.738953    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:25:45.747120 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.738956    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:25:45.747120 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.738958    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:25:45.747120 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.738961    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:25:45.747120 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.738964    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:25:45.747120 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.738967    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:25:45.747120 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.738971    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:25:45.747120 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.738973    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:25:45.747120 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.738976    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:25:45.747120 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.738978    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:25:45.747120 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.738980    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:25:45.747120 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.738983    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:25:45.747120 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.738986    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:25:45.747581 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.738988    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:25:45.747581 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.738991    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:25:45.747581 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.738993    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:25:45.747581 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.738996    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:25:45.747581 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.738998    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:25:45.747581 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739001    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:25:45.747581 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739004    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:25:45.747581 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739006    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:25:45.747581 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739009    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:25:45.747581 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739011    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:25:45.747581 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739013    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:25:45.747581 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739016    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:25:45.747581 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739018    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:25:45.747581 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739022    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:25:45.747581 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739025    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:25:45.747581 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739028    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:25:45.747581 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739030    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:25:45.747581 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739033    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:25:45.747581 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739036    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:25:45.748049 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739039    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:25:45.748049 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739041    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:25:45.748049 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739044    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:25:45.748049 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739046    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:25:45.748049 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739049    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:25:45.748049 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739051    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:25:45.748049 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739055    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:25:45.748049 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739058    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:25:45.748049 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739060    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:25:45.748049 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739063    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:25:45.748049 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739066    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:25:45.748049 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739068    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:25:45.748049 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739071    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:25:45.748049 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739073    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:25:45.748049 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739076    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:25:45.748049 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739078    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:25:45.748049 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739081    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:25:45.748049 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739084    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:25:45.748049 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739086    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:25:45.748049 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739088    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:25:45.748510 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739091    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:25:45.748510 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739093    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:25:45.748510 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739096    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:25:45.748510 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739099    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:25:45.748510 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739101    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:25:45.748510 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739104    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:25:45.748510 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739106    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:25:45.748510 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739109    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:25:45.748510 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739111    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:25:45.748510 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739114    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:25:45.748510 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739116    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:25:45.748510 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739119    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:25:45.748510 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739122    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:25:45.748510 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739124    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:25:45.748510 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739127    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:25:45.748510 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739129    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:25:45.748510 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739132    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:25:45.748510 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739134    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:25:45.748510 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739138    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:25:45.748510 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739141    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:25:45.748990 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739143    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:25:45.748990 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739146    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:25:45.748990 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739148    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:25:45.748990 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739151    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:25:45.748990 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739153    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:25:45.748990 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739156    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:25:45.748990 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739158    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:25:45.748990 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739162    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:25:45.748990 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739166    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:25:45.748990 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739169    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:25:45.748990 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739172    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:25:45.748990 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739174    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:25:45.748990 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739177    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:25:45.748990 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739179    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:25:45.748990 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739182    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:25:45.748990 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739184    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:25:45.748990 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739187    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:25:45.749502 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.739213    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:25:45.749502 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739251    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:25:45.749502 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739255    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:25:45.749502 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739258    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:25:45.749502 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739261    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:25:45.749502 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739263    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:25:45.749502 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739267    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:25:45.749502 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739270    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:25:45.749502 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739275    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:25:45.749502 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739278    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:25:45.749502 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739282    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:25:45.749502 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739285    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:25:45.749502 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739288    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:25:45.749502 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739291    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:25:45.749502 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739295    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:25:45.749883 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739298    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:25:45.749883 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739300    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:25:45.749883 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739303    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:25:45.749883 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739305    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:25:45.749883 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739308    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:25:45.749883 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739310    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:25:45.749883 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739313    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:25:45.749883 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739315    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:25:45.749883 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739318    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:25:45.749883 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739320    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:25:45.749883 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739323    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:25:45.749883 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739325    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:25:45.749883 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739327    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:25:45.749883 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739330    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:25:45.749883 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739333    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:25:45.749883 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739335    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:25:45.749883 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739337    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:25:45.749883 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739340    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:25:45.749883 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739342    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:25:45.749883 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739345    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:25:45.750351 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739347    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:25:45.750351 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739350    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:25:45.750351 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739352    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:25:45.750351 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739356    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:25:45.750351 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739359    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:25:45.750351 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739362    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:25:45.750351 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739364    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:25:45.750351 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739367    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:25:45.750351 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739369    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:25:45.750351 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739371    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:25:45.750351 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739374    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:25:45.750351 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739378    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:25:45.750351 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739418    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:25:45.750351 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739459    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:25:45.750351 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739517    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:25:45.750351 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739521    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:25:45.750351 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739526    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:25:45.750351 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739530    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:25:45.750351 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739534    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:25:45.750351 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739539    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:25:45.750846 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739543    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:25:45.750846 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739548    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:25:45.750846 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739552    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:25:45.750846 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739557    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:25:45.750846 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739561    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:25:45.750846 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739565    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:25:45.750846 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739574    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:25:45.750846 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739579    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:25:45.750846 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739585    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:25:45.750846 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739592    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:25:45.750846 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739598    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:25:45.750846 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739603    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:25:45.750846 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739608    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:25:45.750846 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739613    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:25:45.750846 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739618    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:25:45.750846 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739627    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:25:45.750846 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739632    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:25:45.750846 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739637    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:25:45.750846 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739645    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:25:45.750846 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739650    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:25:45.751314 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739654    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:25:45.751314 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739658    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:25:45.751314 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739662    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:25:45.751314 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739668    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:25:45.751314 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739672    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:25:45.751314 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739678    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:25:45.751314 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739683    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:25:45.751314 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739687    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:25:45.751314 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739691    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:25:45.751314 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739695    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:25:45.751314 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739700    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:25:45.751314 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739709    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:25:45.751314 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739713    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:25:45.751314 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739717    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:25:45.751314 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739722    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:25:45.751314 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739726    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:25:45.751314 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739730    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:25:45.751314 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739734    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:25:45.751314 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739738    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:25:45.751794 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739742    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:25:45.751794 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739746    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:25:45.751794 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739751    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:25:45.751794 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739755    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:25:45.751794 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739758    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:25:45.751794 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739767    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:25:45.751794 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739771    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:25:45.751794 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739777    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:25:45.751794 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739783    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:25:45.751794 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739787    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:25:45.751794 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739791    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:25:45.751794 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739795    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:25:45.751794 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.739800    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:25:45.751794 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.746955    2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 20:25:45.751794 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.746978    2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 20:25:45.751794 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.747567    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 20:25:45.751794 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.747573    2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 20:25:45.751794 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747631    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:25:45.751794 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747637    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:25:45.751794 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747640    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:25:45.752267 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747643    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:25:45.752267 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747646    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:25:45.752267 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747650    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:25:45.752267 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747652    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:25:45.752267 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747655    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:25:45.752267 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747658    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:25:45.752267 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747662    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:25:45.752267 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747666    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:25:45.752267 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747669    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:25:45.752267 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747683    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:25:45.752267 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747686    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:25:45.752267 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747689    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:25:45.752267 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747691    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:25:45.752267 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747694    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:25:45.752267 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747696    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:25:45.752267 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747699    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:25:45.752267 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747702    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:25:45.752267 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747704    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:25:45.752267 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747707    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:25:45.752712 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747709    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:25:45.752712 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747712    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:25:45.752712 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747715    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:25:45.752712 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747717    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:25:45.752712 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747720    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:25:45.752712 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747723    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:25:45.752712 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747725    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:25:45.752712 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747728    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:25:45.752712 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747731    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:25:45.752712 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747734    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:25:45.752712 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747736    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:25:45.752712 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747739    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:25:45.752712 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747741    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:25:45.752712 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747744    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:25:45.752712 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747746    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:25:45.752712 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747749    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:25:45.752712 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747751    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:25:45.752712 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747754    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:25:45.752712 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747756    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:25:45.753171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747759    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:25:45.753171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747761    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:25:45.753171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747763    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:25:45.753171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747766    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:25:45.753171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747769    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:25:45.753171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747771    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:25:45.753171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747774    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:25:45.753171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747776    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:25:45.753171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747778    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:25:45.753171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747781    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:25:45.753171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747784    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:25:45.753171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747786    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:25:45.753171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747788    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:25:45.753171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747791    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:25:45.753171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747793    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:25:45.753171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747796    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:25:45.753171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747798    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:25:45.753171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747800    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:25:45.753171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747803    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:25:45.753171 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747806    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:25:45.753632 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747809    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:25:45.753632 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747812    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:25:45.753632 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747814    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:25:45.753632 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747832    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:25:45.753632 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747836    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:25:45.753632 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747840    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:25:45.753632 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747843    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:25:45.753632 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747846    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:25:45.753632 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747849    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:25:45.753632 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747851    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:25:45.753632 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747854    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:25:45.753632 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747856    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:25:45.753632 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747859    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:25:45.753632 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747862    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:25:45.753632 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747864    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:25:45.753632 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747867    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:25:45.753632 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747869    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:25:45.753632 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747872    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:25:45.753632 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747874    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:25:45.754173 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747877    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:25:45.754173 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747879    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:25:45.754173 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747881    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:25:45.754173 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747884    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:25:45.754173 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747887    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:25:45.754173 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747890    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:25:45.754173 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747893    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:25:45.754173 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747895    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:25:45.754173 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747897    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:25:45.754173 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747900    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:25:45.754173 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747903    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:25:45.754173 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747905    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:25:45.754173 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747908    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:25:45.754173 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747911    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:25:45.754173 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747914    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:25:45.754173 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747917    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:25:45.754173 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747919    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:25:45.754173 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747921    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:25:45.754173 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747924    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:25:45.754173 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747926    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:25:45.754622 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747929    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:25:45.754622 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747931    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:25:45.754622 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747934    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:25:45.754622 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747936    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:25:45.754622 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747939    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:25:45.754622 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.747941    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:25:45.754622 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.747969    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:25:45.754622 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748104    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:25:45.754622 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748108    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:25:45.754622 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748111    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:25:45.754622 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748113    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:25:45.754622 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748116    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:25:45.754622 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748119    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:25:45.754622 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748122    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:25:45.754622 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748125    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:25:45.754622 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748128    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:25:45.755010 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748131    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:25:45.755010 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748135    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:25:45.755010 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748138    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:25:45.755010 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748140    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:25:45.755010 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748143    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:25:45.755010 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748145    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:25:45.755010 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748148    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:25:45.755010 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748151    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:25:45.755010 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748153    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:25:45.755010 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748156    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:25:45.755010 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748158    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:25:45.755010 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748161    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:25:45.755010 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748163    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:25:45.755010 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748166    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:25:45.755010 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748168    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:25:45.755010 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748171    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:25:45.755010 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748174    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:25:45.755010 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748176    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:25:45.755010 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748179    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:25:45.755010 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748181    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:25:45.755473 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748184    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:25:45.755473 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748186    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:25:45.755473 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748188    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:25:45.755473 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748191    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:25:45.755473 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748194    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:25:45.755473 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748196    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:25:45.755473 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748200    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:25:45.755473 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748204    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:25:45.755473 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748207    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:25:45.755473 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748210    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:25:45.755473 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748212    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:25:45.755473 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748215    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:25:45.755473 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748217    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:25:45.755473 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748219    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:25:45.755473 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748223    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:25:45.755473 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748226    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:25:45.755473 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748228    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:25:45.755473 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748230    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:25:45.755473 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748233    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:25:45.755923 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748236    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:25:45.755923 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748238    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:25:45.755923 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748241    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:25:45.755923 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748243    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:25:45.755923 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748246    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:25:45.755923 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748248    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:25:45.755923 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748251    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:25:45.755923 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748253    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:25:45.755923 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748255    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:25:45.755923 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748258    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:25:45.755923 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748260    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:25:45.755923 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748263    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:25:45.755923 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748265    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:25:45.755923 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748268    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:25:45.755923 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748270    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:25:45.755923 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748273    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:25:45.755923 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748275    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:25:45.755923 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748278    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:25:45.755923 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748280    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:25:45.755923 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748283    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:25:45.756387 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748285    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:25:45.756387 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748287    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:25:45.756387 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748290    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:25:45.756387 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748292    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:25:45.756387 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748294    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:25:45.756387 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748297    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:25:45.756387 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748299    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:25:45.756387 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748302    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:25:45.756387 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748305    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:25:45.756387 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748308    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:25:45.756387 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748310    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:25:45.756387 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748313    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:25:45.756387 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748315    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:25:45.756387 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748318    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:25:45.756387 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748320    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:25:45.756387 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748323    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:25:45.756387 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748325    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:25:45.756387 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748328    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:25:45.756387 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748330    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:25:45.756387 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748333    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:25:45.756859 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748335    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:25:45.756859 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748338    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:25:45.756859 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748340    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:25:45.756859 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748342    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:25:45.756859 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748345    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:25:45.756859 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748348    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:25:45.756859 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748350    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:25:45.756859 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748353    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:25:45.756859 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748355    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:25:45.756859 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748357    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:25:45.756859 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748360    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:25:45.756859 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748362    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:25:45.756859 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748364    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:25:45.756859 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748367    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:25:45.756859 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748369    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:25:45.756859 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748372    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:25:45.756859 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748374    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:25:45.756859 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748376    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:25:45.757276 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.748399    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:25:45.757276 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748428    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:25:45.757276 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748432    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:25:45.757276 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748436    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:25:45.757276 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748438    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:25:45.757276 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748441    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:25:45.757276 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748444    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:25:45.757276 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748447    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:25:45.757276 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748449    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:25:45.757276 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748452    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:25:45.757276 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748454    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:25:45.757276 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748456    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:25:45.757276 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748459    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:25:45.757276 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748461    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:25:45.757276 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748464    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:25:45.757276 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748466    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:25:45.757646 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748468    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:25:45.757646 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748472    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:25:45.757646 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748476    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:25:45.757646 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748479    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:25:45.757646 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748482    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:25:45.757646 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748485    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:25:45.757646 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748487    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:25:45.757646 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748490    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:25:45.757646 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748492    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:25:45.757646 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748494    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:25:45.757646 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748497    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:25:45.757646 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748499    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:25:45.757646 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748502    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:25:45.757646 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748504    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:25:45.757646 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748507    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:25:45.757646 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748509    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:25:45.757646 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748512    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:25:45.757646 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748515    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:25:45.757646 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748517    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:25:45.758134 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748520    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:25:45.758134 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748523    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:25:45.758134 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748525    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:25:45.758134 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748528    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:25:45.758134 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748531    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:25:45.758134 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748533    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:25:45.758134 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748535    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:25:45.758134 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748538    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:25:45.758134 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748542    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:25:45.758134 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748544    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:25:45.758134 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748547    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:25:45.758134 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748549    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:25:45.758134 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748551    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:25:45.758134 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748554    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:25:45.758134 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748556    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:25:45.758134 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748559    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:25:45.758134 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748561    2575 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:25:45.758134 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748564    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:25:45.758134 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748566    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:25:45.758134 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748569    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:25:45.758617 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748571    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:25:45.758617 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748573    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:25:45.758617 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748576    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:25:45.758617 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748578    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:25:45.758617 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748580    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:25:45.758617 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748583    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:25:45.758617 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748585    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:25:45.758617 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748587    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:25:45.758617 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748590    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:25:45.758617 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748592    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:25:45.758617 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748595    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:25:45.758617 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748597    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:25:45.758617 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748599    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:25:45.758617 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748602    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:25:45.758617 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748612    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:25:45.758617 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748615    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:25:45.758617 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748618    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:25:45.758617 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748625    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:25:45.758617 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748628    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:25:45.758617 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748630    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:25:45.759105 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748632    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:25:45.759105 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748635    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:25:45.759105 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748637    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:25:45.759105 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748640    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:25:45.759105 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748642    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:25:45.759105 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748644    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:25:45.759105 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748647    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:25:45.759105 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748649    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:25:45.759105 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748652    2575 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:25:45.759105 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748654    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:25:45.759105 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748657    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:25:45.759105 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748659    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:25:45.759105 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748661    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:25:45.759105 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748664    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:25:45.759105 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748666    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:25:45.759105 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748669    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:25:45.759105 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748671    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:25:45.759105 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748674    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:25:45.759105 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748678    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:25:45.759577 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748682    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:25:45.759577 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748686    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:25:45.759577 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748689    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:25:45.759577 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748692    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:25:45.759577 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748694    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:25:45.759577 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748697    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:25:45.759577 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748699    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:25:45.759577 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748702    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:25:45.759577 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748705    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:25:45.759577 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748707    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:25:45.759577 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748710    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:25:45.759577 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748712    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:25:45.759577 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:45.748714    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:25:45.759577 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.748769    2575 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 20:25:45.759577 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.751577    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 20:25:45.759577 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.752486    2575 server.go:1014] "Starting client certificate rotation"
Oct 30 20:25:45.759577 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.752583    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 20:25:45.759577 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.752645    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 20:25:45.760001 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.754472    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 20:25:45.760001 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.758270    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 20:25:45.771256 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.771239    2575 log.go:25] "Validated CRI v1 runtime API"
Oct 30 20:25:45.774922 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.774904    2575 log.go:25] "Validated CRI v1 image API"
Oct 30 20:25:45.775726 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.775691    2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 20:25:45.778256 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.778238    2575 fs.go:135] Filesystem UUIDs: map[73df03d4-9128-4efa-8ca8-0c4e682c0a29:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8a73f35e-5ef0-43e1-88f3-50fa93954dcd:/dev/nvme0n1p3]
Oct 30 20:25:45.778313 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.778255    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 20:25:45.784035 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.783923    2575 manager.go:217] Machine: {Timestamp:2026-10-30 20:25:45.782479389 +0000 UTC m=+0.353624459 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104253 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bd3fa94c1a01fdf528a40e952f581 SystemUUID:ec2bd3fa-94c1-a01f-df52-8a40e952f581 BootID:d2547233-e9df-416f-9af6-3c280c55b54c Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:66:29:2f:86:cf Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:66:29:2f:86:cf Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e2:60:f4:bd:02:47 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 20:25:45.785738 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.785727    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 20:25:45.785840 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.785804    2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 20:25:45.789837 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.789471    2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 20:25:45.789837 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.789515    2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-149.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 20:25:45.789837 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.789694    2575 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 20:25:45.789837 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.789707    2575 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 20:25:45.789837 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.789744    2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 20:25:45.789837 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.789765    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 20:25:45.791058 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.791045    2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 20:25:45.795026 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.795014    2575 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 20:25:45.795150 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.795141    2575 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 20:25:45.795843 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.795832    2575 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 20:25:45.795872 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.795848    2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 20:25:45.795872 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.795859    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 20:25:45.795872 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.795868    2575 kubelet.go:405] "Adding apiserver pod source"
Oct 30 20:25:45.795992 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.795876    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 20:25:45.796177 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.796165    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 20:25:45.798428 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.798405    2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 20:25:45.804574 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.804560    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 20:25:45.805468 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.805456    2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 20:25:45.805511 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.805486    2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 20:25:45.805633 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.805624    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 20:25:45.805665 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.805638    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 20:25:45.805665 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.805645    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 20:25:45.805665 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.805650    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 20:25:45.805665 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.805655    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 20:25:45.805665 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.805661    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 20:25:45.806327 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.806318    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 20:25:45.806357 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.806327    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 20:25:45.806357 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.806335    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 20:25:45.806357 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.806340    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 20:25:45.806357 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.806354    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 20:25:45.806471 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.806363    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 20:25:45.807073 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.807065    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 20:25:45.807103 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.807076    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 20:25:45.810988 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.810976    2575 server.go:1263] "Started kubelet"
Oct 30 20:25:45.811089 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.811057    2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 20:25:45.811178 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.811113    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 20:25:45.811213 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.811200    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 20:25:45.811674 ip-10-0-139-149 systemd[1]: Started Kubernetes Kubelet.
Oct 30 20:25:45.813938 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.813921    2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 20:25:45.814267 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.814255    2575 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 20:25:45.817356 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:45.817340    2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 20:25:45.819431 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.819412    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 20:25:45.820713 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.820697    2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 20:25:45.820858 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:45.820838    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-149.us-west-1.compute.internal\" not found"
Oct 30 20:25:45.821492 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.821475    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 20:25:45.821492 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.821478    2575 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 20:25:45.821647 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.821500    2575 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 20:25:45.821647 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.821553    2575 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 20:25:45.821647 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.821563    2575 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 20:25:45.822070 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.822057    2575 factory.go:55] Registering systemd factory
Oct 30 20:25:45.822070 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.822072    2575 factory.go:223] Registration of the systemd container factory successfully
Oct 30 20:25:45.822316 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.822302    2575 factory.go:153] Registering CRI-O factory
Oct 30 20:25:45.822316 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.822315    2575 factory.go:223] Registration of the crio container factory successfully
Oct 30 20:25:45.822428 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.822368    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 20:25:45.822428 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.822389    2575 factory.go:103] Registering Raw factory
Oct 30 20:25:45.822428 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.822416    2575 manager.go:1199] Started watching for new ooms in manager
Oct 30 20:25:45.824049 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.824030    2575 manager.go:319] Starting recovery of all containers
Oct 30 20:25:45.836417 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.836395    2575 manager.go:324] Recovery completed
Oct 30 20:25:45.840420 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.840403    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:25:45.841845 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.841810    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-149.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:25:45.841908 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.841867    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-149.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:25:45.841908 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.841882    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-149.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:25:45.842379 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.842367    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:25:45.842431 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.842367    2575 cpu_manager.go:226] "Starting" policy="none"
Oct 30 20:25:45.842431 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.842396    2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 20:25:45.842431 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.842413    2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 20:25:45.843009 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.842996    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-149.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:25:45.843062 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.843021    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-149.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:25:45.843062 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.843031    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-149.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:25:45.844589 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.844572    2575 policy_none.go:50] "Start"
Oct 30 20:25:45.844589 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.844587    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 20:25:45.844722 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.844598    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 20:25:45.846530 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.846518    2575 policy_none.go:44] "Start"
Oct 30 20:25:45.881477 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.881338    2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 20:25:45.885282 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.885268    2575 manager.go:341] "Starting Device Plugin manager"
Oct 30 20:25:45.885936 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:45.885922    2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 20:25:45.886014 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.885939    2575 server.go:87] "Starting device plugin registration server"
Oct 30 20:25:45.886173 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.886160    2575 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 20:25:45.886252 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.886176    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 20:25:45.886302 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.886265    2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 20:25:45.886469 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.886446    2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 20:25:45.886469 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.886469    2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 20:25:45.887157 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:45.887125    2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 20:25:45.887230 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:45.887179    2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-149.us-west-1.compute.internal\" not found"
Oct 30 20:25:45.953017 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.952981    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 20:25:45.954632 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.953620    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-149.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 20:25:45.954632 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:45.953493    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-149.us-west-1.compute.internal.18e36879111bba6b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-149.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-149.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 20:25:45.810950763 +0000 UTC m=+0.382095833,LastTimestamp:2026-10-30 20:25:45.810950763 +0000 UTC m=+0.382095833,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-149.us-west-1.compute.internal,}"
Oct 30 20:25:45.954632 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.954387    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 20:25:45.954632 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.954471    2575 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 20:25:45.954632 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.954496    2575 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 20:25:45.954632 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:45.954535    2575 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 20:25:45.987156 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.987100    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:25:45.987957 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.987938    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-149.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:25:45.988055 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.987969    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-149.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:25:45.988055 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.987990    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-149.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:25:45.988055 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:45.988018    2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:25:46.001683 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:46.001628    2575 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-149.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 20:25:46.028275 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.028255    2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 20:25:46.028342 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.028332    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gg6xc"
Oct 30 20:25:46.055386 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.055362    2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-149.us-west-1.compute.internal"]
Oct 30 20:25:46.055445 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.055434    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:25:46.056489 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.056473    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-149.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:25:46.056568 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.056501    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-149.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:25:46.056568 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.056511    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-149.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:25:46.059056 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.059044    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:25:46.059300 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.059280    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:25:46.059372 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.059337    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:25:46.059925 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.059893    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-149.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:25:46.059925 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.059927    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-149.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:25:46.060049 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.059944    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-149.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:25:46.060105 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.060092    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-149.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:25:46.060157 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.060119    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-149.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:25:46.060157 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.060130    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-149.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:25:46.061840 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.061811    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:25:46.062016 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.061852    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:25:46.062543 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.062522    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-149.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:25:46.062633 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.062556    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-149.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:25:46.062633 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.062566    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-149.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:25:46.083805 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:46.083781    2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-149.us-west-1.compute.internal\" not found" node="ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:25:46.086137 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:46.086122    2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-149.us-west-1.compute.internal\" not found" node="ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:25:46.109835 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.109800    2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:25:46.111103 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.111088    2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:25:46.208944 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.208900    2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:25:46.209929 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.209908    2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:25:46.210010 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:46.209948    2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-149.us-west-1.compute.internal\": node \"ip-10-0-139-149.us-west-1.compute.internal\" not found"
Oct 30 20:25:46.210509 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.210491    2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:25:46.221102 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.221084    2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:25:46.222420 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.222403    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9d9f66ca5c88277c8c76e3090f63988d-config\") pod \"kube-apiserver-proxy-ip-10-0-139-149.us-west-1.compute.internal\" (UID: \"9d9f66ca5c88277c8c76e3090f63988d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:25:46.222487 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.222463    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c591170baa3135b88257a110e786e9de-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal\" (UID: \"c591170baa3135b88257a110e786e9de\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:25:46.222520 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.222506    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c591170baa3135b88257a110e786e9de-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal\" (UID: \"c591170baa3135b88257a110e786e9de\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:25:46.323141 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.323116    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9d9f66ca5c88277c8c76e3090f63988d-config\") pod \"kube-apiserver-proxy-ip-10-0-139-149.us-west-1.compute.internal\" (UID: \"9d9f66ca5c88277c8c76e3090f63988d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:25:46.323237 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.323147    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c591170baa3135b88257a110e786e9de-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal\" (UID: \"c591170baa3135b88257a110e786e9de\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:25:46.323237 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.323181    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c591170baa3135b88257a110e786e9de-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal\" (UID: \"c591170baa3135b88257a110e786e9de\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:25:46.323237 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.323228    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c591170baa3135b88257a110e786e9de-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal\" (UID: \"c591170baa3135b88257a110e786e9de\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:25:46.323328 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.323241    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9d9f66ca5c88277c8c76e3090f63988d-config\") pod \"kube-apiserver-proxy-ip-10-0-139-149.us-west-1.compute.internal\" (UID: \"9d9f66ca5c88277c8c76e3090f63988d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:25:46.323328 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.323241    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c591170baa3135b88257a110e786e9de-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal\" (UID: \"c591170baa3135b88257a110e786e9de\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:25:46.335356 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.335338    2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:25:46.336513 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.336502    2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:25:46.386201 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.386183    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:25:46.389280 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.389267    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:25:46.440557 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.440529    2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:25:46.752914 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.752840    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 20:25:46.753381 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.752959    2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 20:25:46.753381 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.753030    2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 20:25:46.753381 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:46.753031    2575 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a387d27a2dc53463097c2534c2d2347e-212d01575edb5520.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-139-149.us-west-1.compute.internal.18e3687912f3c30a\": read tcp 10.0.139.149:47272->18.116.137.94:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-139-149.us-west-1.compute.internal.18e3687912f3c30a  default   4790 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-149.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-149.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-149.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 20:25:45 +0000 UTC,LastTimestamp:2026-10-30 20:25:45.987994046 +0000 UTC m=+0.559139116,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-149.us-west-1.compute.internal,}"
Oct 30 20:25:46.796734 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.796712    2575 apiserver.go:52] "Watching apiserver"
Oct 30 20:25:46.819946 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:46.819927    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 20:25:47.033005 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.032915    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 20:20:46 +0000 UTC" deadline="2028-05-10 16:49:11.83185237 +0000 UTC"
Oct 30 20:25:47.033005 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.032951    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13388h23m24.798904301s"
Oct 30 20:25:47.057368 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.057343    2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 20:25:47.057656 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.057633    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-kwm7g","kube-system/konnectivity-agent-4khkm","openshift-dns/node-resolver-c8422","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95","openshift-image-registry/node-ca-rdqrc","openshift-ovn-kubernetes/ovnkube-node-q4f4h","kube-system/global-pull-secret-syncer-bqh9l","kube-system/kube-apiserver-proxy-ip-10-0-139-149.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-59qcr","openshift-multus/multus-additional-cni-plugins-qxwlb","openshift-multus/network-metrics-daemon-zcslg","openshift-multus/multus-2b77p","openshift-network-diagnostics/network-check-target-pfvnn"]
Oct 30 20:25:47.060020 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.060003    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:25:47.061924 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.061893    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfvnn"
Oct 30 20:25:47.062027 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:47.061973    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfvnn" podUID="c9732ee5-c72e-44c6-9917-49cc9f81ffe0"
Oct 30 20:25:47.063879 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.063862    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.066027 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.066013    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.068051 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.068037    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:25:47.070082 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.070067    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rdqrc"
Oct 30 20:25:47.072267 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.072253    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kwm7g"
Oct 30 20:25:47.074955 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.074924    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:25:47.075111 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:47.075073    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bqh9l" podUID="88d9611e-c7ed-490b-b401-530d1649c873"
Oct 30 20:25:47.077642 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.077620    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.079905 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.079889    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:25:47.079980 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:47.079940    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcslg" podUID="214f4f91-4d6e-46e0-ac1f-d23935cd7215"
Oct 30 20:25:47.082161 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.082142    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c8422"
Oct 30 20:25:47.084250 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.084233    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4khkm"
Oct 30 20:25:47.110005 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.109988    2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 20:25:47.110099 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.110040    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jz4mh"
Oct 30 20:25:47.114603 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.114585    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 20:25:47.114713 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.114689    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 20:25:47.114777 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.114711    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 20:25:47.114777 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.114728    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 20:25:47.114777 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.114755    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ld6lr\""
Oct 30 20:25:47.114777 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.114764    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 20:25:47.116121 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.116107    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-cnj9l\""
Oct 30 20:25:47.116696 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.116681    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 20:25:47.118294 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.118278    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 20:25:47.118587 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.118569    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-47trg\""
Oct 30 20:25:47.118587 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.118571    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 20:25:47.118870 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.118847    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 20:25:47.118994 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.118895    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 20:25:47.119035 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.119002    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 20:25:47.119069 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.119046    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 20:25:47.122936 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.122920    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 20:25:47.127473 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.127452    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-host-run-multus-certs\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.127552 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.127480    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-run-ovn-kubernetes\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.127552 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.127499    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/88d9611e-c7ed-490b-b401-530d1649c873-dbus\") pod \"global-pull-secret-syncer-bqh9l\" (UID: \"88d9611e-c7ed-490b-b401-530d1649c873\") " pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:25:47.127552 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.127524    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-kubernetes\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.127643 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.127561    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/89ed3d9f-e057-40a3-add4-d6374380be1f-agent-certs\") pod \"konnectivity-agent-4khkm\" (UID: \"89ed3d9f-e057-40a3-add4-d6374380be1f\") " pod="kube-system/konnectivity-agent-4khkm"
Oct 30 20:25:47.127643 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.127598    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:25:47.127700 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.127654    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-slash\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.127700 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.127691    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-var-lib-openvswitch\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.127789 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.127712    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llcbk\" (UniqueName: \"kubernetes.io/projected/8b7865f8-cfd2-4938-8621-92b63faee7af-kube-api-access-llcbk\") pod \"node-ca-rdqrc\" (UID: \"8b7865f8-cfd2-4938-8621-92b63faee7af\") " pod="openshift-image-registry/node-ca-rdqrc"
Oct 30 20:25:47.127789 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.127727    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w5vd\" (UniqueName: \"kubernetes.io/projected/88178646-9467-4225-b362-578f31454922-kube-api-access-2w5vd\") pod \"node-resolver-c8422\" (UID: \"88178646-9467-4225-b362-578f31454922\") " pod="openshift-dns/node-resolver-c8422"
Oct 30 20:25:47.127789 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.127747    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:25:47.127906 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.127784    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-host-var-lib-cni-multus\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.127906 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.127814    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-log-socket\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.127906 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.127864    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-cni-bin\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.127906 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.127880    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-sysconfig\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.127906 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.127902    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-lib-modules\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.128048 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.127921    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-cni-binary-copy\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.128048 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.127974    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea45e6d2-1a47-4956-b652-19b428800f42-socket-dir\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:25:47.128048 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128010    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea45e6d2-1a47-4956-b652-19b428800f42-registration-dir\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:25:47.128048 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128035    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b7865f8-cfd2-4938-8621-92b63faee7af-host\") pod \"node-ca-rdqrc\" (UID: \"8b7865f8-cfd2-4938-8621-92b63faee7af\") " pod="openshift-image-registry/node-ca-rdqrc"
Oct 30 20:25:47.128156 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128052    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/88d9611e-c7ed-490b-b401-530d1649c873-kubelet-config\") pod \"global-pull-secret-syncer-bqh9l\" (UID: \"88d9611e-c7ed-490b-b401-530d1649c873\") " pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:25:47.128156 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128066    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-system-cni-dir\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:25:47.128156 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128080    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4vlt\" (UniqueName: \"kubernetes.io/projected/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-kube-api-access-l4vlt\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:25:47.128156 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128096    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb792\" (UniqueName: \"kubernetes.io/projected/c9732ee5-c72e-44c6-9917-49cc9f81ffe0-kube-api-access-bb792\") pod \"network-check-target-pfvnn\" (UID: \"c9732ee5-c72e-44c6-9917-49cc9f81ffe0\") " pod="openshift-network-diagnostics/network-check-target-pfvnn"
Oct 30 20:25:47.128156 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128141    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-run-systemd\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.128288 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128163    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjn87\" (UniqueName: \"kubernetes.io/projected/80cae79e-b60f-4efe-bb58-e77753a457e5-kube-api-access-kjn87\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.128288 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128180    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/88d9611e-c7ed-490b-b401-530d1649c873-original-pull-secret\") pod \"global-pull-secret-syncer-bqh9l\" (UID: \"88d9611e-c7ed-490b-b401-530d1649c873\") " pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:25:47.128288 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128194    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:25:47.128288 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128234    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.128288 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128254    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/80cae79e-b60f-4efe-bb58-e77753a457e5-ovnkube-config\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.128288 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128273    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwzfx\" (UniqueName: \"kubernetes.io/projected/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-kube-api-access-cwzfx\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.128446 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128308    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-multus-conf-dir\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.128446 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128335    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea45e6d2-1a47-4956-b652-19b428800f42-etc-selinux\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:25:47.128446 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128356    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-host\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.128446 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128374    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx2vz\" (UniqueName: \"kubernetes.io/projected/214f4f91-4d6e-46e0-ac1f-d23935cd7215-kube-api-access-nx2vz\") pod \"network-metrics-daemon-zcslg\" (UID: \"214f4f91-4d6e-46e0-ac1f-d23935cd7215\") " pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:25:47.128446 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128400    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/88178646-9467-4225-b362-578f31454922-hosts-file\") pod \"node-resolver-c8422\" (UID: \"88178646-9467-4225-b362-578f31454922\") " pod="openshift-dns/node-resolver-c8422"
Oct 30 20:25:47.128446 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128415    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88178646-9467-4225-b362-578f31454922-tmp-dir\") pod \"node-resolver-c8422\" (UID: \"88178646-9467-4225-b362-578f31454922\") " pod="openshift-dns/node-resolver-c8422"
Oct 30 20:25:47.128446 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128446    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-cnibin\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:25:47.128629 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128479    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-cni-binary-copy\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:25:47.128629 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128496    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-run-openvswitch\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.128629 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128516    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-systemd\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.128629 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128549    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-multus-cni-dir\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.128629 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128572    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-host-var-lib-cni-bin\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.128629 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128591    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-systemd-units\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.128629 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128623    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea45e6d2-1a47-4956-b652-19b428800f42-sys-fs\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:25:47.128810 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128646    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgvld\" (UniqueName: \"kubernetes.io/projected/111798c9-6959-4dcb-9a8a-db3037f7d662-kube-api-access-pgvld\") pod \"iptables-alerter-kwm7g\" (UID: \"111798c9-6959-4dcb-9a8a-db3037f7d662\") " pod="openshift-network-operator/iptables-alerter-kwm7g"
Oct 30 20:25:47.128810 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128660    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-sys\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.128810 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128677    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-var-lib-kubelet\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.128810 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128707    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-host-run-netns\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.128810 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128727    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkqqp\" (UniqueName: \"kubernetes.io/projected/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-kube-api-access-mkqqp\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.128810 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128746    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-run-ovn\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.128810 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128766    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/80cae79e-b60f-4efe-bb58-e77753a457e5-env-overrides\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.128810 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128788    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-sysctl-conf\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.129080 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128812    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-tmp\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.129080 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128852    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/214f4f91-4d6e-46e0-ac1f-d23935cd7215-metrics-certs\") pod \"network-metrics-daemon-zcslg\" (UID: \"214f4f91-4d6e-46e0-ac1f-d23935cd7215\") " pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:25:47.129080 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128874    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-system-cni-dir\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.129080 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128889    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-cnibin\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.129080 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128910    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-host-var-lib-kubelet\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.129080 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128934    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-modprobe-d\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.129080 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128956    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-sysctl-d\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.129080 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128979    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-kubelet\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.129080 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.128998    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/80cae79e-b60f-4efe-bb58-e77753a457e5-ovnkube-script-lib\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.129080 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.129018    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea45e6d2-1a47-4956-b652-19b428800f42-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:25:47.129080 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.129041    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5zcp\" (UniqueName: \"kubernetes.io/projected/ea45e6d2-1a47-4956-b652-19b428800f42-kube-api-access-h5zcp\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:25:47.129080 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.129060    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-node-log\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.129080 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.129080    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-run\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.129414 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.129097    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-multus-socket-dir-parent\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.129414 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.129123    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-host-run-k8s-cni-cncf-io\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.129414 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.129148    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-multus-daemon-config\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.129414 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.129164    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-etc-kubernetes\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.129414 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.129177    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-cni-netd\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.129414 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.129190    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-tuned\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.129414 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.129212    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/89ed3d9f-e057-40a3-add4-d6374380be1f-konnectivity-ca\") pod \"konnectivity-agent-4khkm\" (UID: \"89ed3d9f-e057-40a3-add4-d6374380be1f\") " pod="kube-system/konnectivity-agent-4khkm"
Oct 30 20:25:47.129414 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.129230    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea45e6d2-1a47-4956-b652-19b428800f42-device-dir\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:25:47.129414 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.129254    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/111798c9-6959-4dcb-9a8a-db3037f7d662-host-slash\") pod \"iptables-alerter-kwm7g\" (UID: \"111798c9-6959-4dcb-9a8a-db3037f7d662\") " pod="openshift-network-operator/iptables-alerter-kwm7g"
Oct 30 20:25:47.129414 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.129267    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-hostroot\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.129414 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.129280    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-run-netns\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.129414 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.129299    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-etc-openvswitch\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.129414 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.129314    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/80cae79e-b60f-4efe-bb58-e77753a457e5-ovn-node-metrics-cert\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.129414 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.129326    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8b7865f8-cfd2-4938-8621-92b63faee7af-serviceca\") pod \"node-ca-rdqrc\" (UID: \"8b7865f8-cfd2-4938-8621-92b63faee7af\") " pod="openshift-image-registry/node-ca-rdqrc"
Oct 30 20:25:47.129414 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.129339    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/111798c9-6959-4dcb-9a8a-db3037f7d662-iptables-alerter-script\") pod \"iptables-alerter-kwm7g\" (UID: \"111798c9-6959-4dcb-9a8a-db3037f7d662\") " pod="openshift-network-operator/iptables-alerter-kwm7g"
Oct 30 20:25:47.159783 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.159757    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 20:25:47.159783 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.159759    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 20:25:47.160041 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.159801    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bx2zg\""
Oct 30 20:25:47.160041 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.159945    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 20:25:47.160041 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.159980    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 20:25:47.160041 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.160002    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 20:25:47.160232 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.160046    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 20:25:47.160232 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.160072    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 20:25:47.160232 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.160110    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 20:25:47.162126 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.162099    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:25:47.162219 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.162133    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:25:47.162219 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.162180    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-52gtb\""
Oct 30 20:25:47.163206 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.163194    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wczr5\""
Oct 30 20:25:47.163531 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.163518    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 20:25:47.163731 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.163712    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qmclr\""
Oct 30 20:25:47.163812 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.163722    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9wdmm\""
Oct 30 20:25:47.163812 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.163774    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 20:25:47.163812 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.163792    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 20:25:47.163954 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.163845    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 20:25:47.163954 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.163895    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fngvs\""
Oct 30 20:25:47.164067 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.164045    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 20:25:47.230258 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230230    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-kubelet\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.230258 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230260    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/80cae79e-b60f-4efe-bb58-e77753a457e5-ovnkube-script-lib\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.230453 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230280    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea45e6d2-1a47-4956-b652-19b428800f42-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:25:47.230453 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230295    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5zcp\" (UniqueName: \"kubernetes.io/projected/ea45e6d2-1a47-4956-b652-19b428800f42-kube-api-access-h5zcp\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:25:47.230453 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230318    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-node-log\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.230453 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230335    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-run\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.230453 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230349    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-kubelet\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.230453 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230391    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea45e6d2-1a47-4956-b652-19b428800f42-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:25:47.230453 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230407    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-multus-socket-dir-parent\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.230453 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230413    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-node-log\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.230453 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230443    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-host-run-k8s-cni-cncf-io\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.230854 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230465    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-run\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.230854 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230471    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-multus-daemon-config\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.230854 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230499    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-multus-socket-dir-parent\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.230854 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230503    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-etc-kubernetes\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.230854 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230532    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-etc-kubernetes\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.230854 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230530    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-host-run-k8s-cni-cncf-io\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.230854 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230544    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-cni-netd\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.230854 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230571    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-tuned\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.230854 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230574    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-cni-netd\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.230854 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230597    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/89ed3d9f-e057-40a3-add4-d6374380be1f-konnectivity-ca\") pod \"konnectivity-agent-4khkm\" (UID: \"89ed3d9f-e057-40a3-add4-d6374380be1f\") " pod="kube-system/konnectivity-agent-4khkm"
Oct 30 20:25:47.230854 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230621    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea45e6d2-1a47-4956-b652-19b428800f42-device-dir\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:25:47.230854 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230644    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/111798c9-6959-4dcb-9a8a-db3037f7d662-host-slash\") pod \"iptables-alerter-kwm7g\" (UID: \"111798c9-6959-4dcb-9a8a-db3037f7d662\") " pod="openshift-network-operator/iptables-alerter-kwm7g"
Oct 30 20:25:47.230854 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230658    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-hostroot\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.230854 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230675    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-run-netns\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.230854 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230721    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-run-netns\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.230854 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230722    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/111798c9-6959-4dcb-9a8a-db3037f7d662-host-slash\") pod \"iptables-alerter-kwm7g\" (UID: \"111798c9-6959-4dcb-9a8a-db3037f7d662\") " pod="openshift-network-operator/iptables-alerter-kwm7g"
Oct 30 20:25:47.230854 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230741    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-etc-openvswitch\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.230854 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230737    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-hostroot\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.231676 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230770    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/80cae79e-b60f-4efe-bb58-e77753a457e5-ovn-node-metrics-cert\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.231676 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230780    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea45e6d2-1a47-4956-b652-19b428800f42-device-dir\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:25:47.231676 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230837    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-etc-openvswitch\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.231676 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230805    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8b7865f8-cfd2-4938-8621-92b63faee7af-serviceca\") pod \"node-ca-rdqrc\" (UID: \"8b7865f8-cfd2-4938-8621-92b63faee7af\") " pod="openshift-image-registry/node-ca-rdqrc"
Oct 30 20:25:47.231676 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230875    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/111798c9-6959-4dcb-9a8a-db3037f7d662-iptables-alerter-script\") pod \"iptables-alerter-kwm7g\" (UID: \"111798c9-6959-4dcb-9a8a-db3037f7d662\") " pod="openshift-network-operator/iptables-alerter-kwm7g"
Oct 30 20:25:47.231676 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230925    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-host-run-multus-certs\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.231676 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230940    2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 20:25:47.231676 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230953    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-run-ovn-kubernetes\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.231676 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.230973    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/80cae79e-b60f-4efe-bb58-e77753a457e5-ovnkube-script-lib\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.231676 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.231042    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-host-run-multus-certs\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.231676 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.231097    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-multus-daemon-config\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.231676 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.231135    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/89ed3d9f-e057-40a3-add4-d6374380be1f-konnectivity-ca\") pod \"konnectivity-agent-4khkm\" (UID: \"89ed3d9f-e057-40a3-add4-d6374380be1f\") " pod="kube-system/konnectivity-agent-4khkm"
Oct 30 20:25:47.231676 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.231139    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-run-ovn-kubernetes\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.231676 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.231257    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8b7865f8-cfd2-4938-8621-92b63faee7af-serviceca\") pod \"node-ca-rdqrc\" (UID: \"8b7865f8-cfd2-4938-8621-92b63faee7af\") " pod="openshift-image-registry/node-ca-rdqrc"
Oct 30 20:25:47.231676 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.231510    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/111798c9-6959-4dcb-9a8a-db3037f7d662-iptables-alerter-script\") pod \"iptables-alerter-kwm7g\" (UID: \"111798c9-6959-4dcb-9a8a-db3037f7d662\") " pod="openshift-network-operator/iptables-alerter-kwm7g"
Oct 30 20:25:47.231676 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.231574    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/88d9611e-c7ed-490b-b401-530d1649c873-dbus\") pod \"global-pull-secret-syncer-bqh9l\" (UID: \"88d9611e-c7ed-490b-b401-530d1649c873\") " pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:25:47.231676 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.231600    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-kubernetes\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.231676 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.231625    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/89ed3d9f-e057-40a3-add4-d6374380be1f-agent-certs\") pod \"konnectivity-agent-4khkm\" (UID: \"89ed3d9f-e057-40a3-add4-d6374380be1f\") " pod="kube-system/konnectivity-agent-4khkm"
Oct 30 20:25:47.232466 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.231651    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:25:47.232466 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.231676    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-slash\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.232466 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.231681    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-kubernetes\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.232466 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.231721    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/88d9611e-c7ed-490b-b401-530d1649c873-dbus\") pod \"global-pull-secret-syncer-bqh9l\" (UID: \"88d9611e-c7ed-490b-b401-530d1649c873\") " pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:25:47.232466 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.231750    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-slash\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.232466 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.231778    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-var-lib-openvswitch\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.232466 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.231805    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llcbk\" (UniqueName: \"kubernetes.io/projected/8b7865f8-cfd2-4938-8621-92b63faee7af-kube-api-access-llcbk\") pod \"node-ca-rdqrc\" (UID: \"8b7865f8-cfd2-4938-8621-92b63faee7af\") " pod="openshift-image-registry/node-ca-rdqrc"
Oct 30 20:25:47.232466 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.231847    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w5vd\" (UniqueName: \"kubernetes.io/projected/88178646-9467-4225-b362-578f31454922-kube-api-access-2w5vd\") pod \"node-resolver-c8422\" (UID: \"88178646-9467-4225-b362-578f31454922\") " pod="openshift-dns/node-resolver-c8422"
Oct 30 20:25:47.232466 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.231871    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:25:47.232466 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.231895    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-host-var-lib-cni-multus\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.232466 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.231918    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-log-socket\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.232466 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.231939    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-cni-bin\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.232466 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.231962    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-sysconfig\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.232466 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232003    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-lib-modules\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.232466 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232029    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-cni-binary-copy\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.232466 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232054    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea45e6d2-1a47-4956-b652-19b428800f42-socket-dir\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:25:47.232466 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232077    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea45e6d2-1a47-4956-b652-19b428800f42-registration-dir\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:25:47.233195 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232090    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:25:47.233195 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232105    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b7865f8-cfd2-4938-8621-92b63faee7af-host\") pod \"node-ca-rdqrc\" (UID: \"8b7865f8-cfd2-4938-8621-92b63faee7af\") " pod="openshift-image-registry/node-ca-rdqrc"
Oct 30 20:25:47.233195 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232138    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/88d9611e-c7ed-490b-b401-530d1649c873-kubelet-config\") pod \"global-pull-secret-syncer-bqh9l\" (UID: \"88d9611e-c7ed-490b-b401-530d1649c873\") " pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:25:47.233195 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232148    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b7865f8-cfd2-4938-8621-92b63faee7af-host\") pod \"node-ca-rdqrc\" (UID: \"8b7865f8-cfd2-4938-8621-92b63faee7af\") " pod="openshift-image-registry/node-ca-rdqrc"
Oct 30 20:25:47.233195 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.231870    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-var-lib-openvswitch\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.233195 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232163    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-system-cni-dir\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:25:47.233195 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232193    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-system-cni-dir\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:25:47.233195 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232254    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-log-socket\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.233195 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232263    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/88d9611e-c7ed-490b-b401-530d1649c873-kubelet-config\") pod \"global-pull-secret-syncer-bqh9l\" (UID: \"88d9611e-c7ed-490b-b401-530d1649c873\") " pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:25:47.233195 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232298    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-cni-bin\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.233195 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232192    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4vlt\" (UniqueName: \"kubernetes.io/projected/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-kube-api-access-l4vlt\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:25:47.233195 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232312    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-sysconfig\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.233195 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232325    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb792\" (UniqueName: \"kubernetes.io/projected/c9732ee5-c72e-44c6-9917-49cc9f81ffe0-kube-api-access-bb792\") pod \"network-check-target-pfvnn\" (UID: \"c9732ee5-c72e-44c6-9917-49cc9f81ffe0\") " pod="openshift-network-diagnostics/network-check-target-pfvnn"
Oct 30 20:25:47.233195 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232344    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-run-systemd\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.233195 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232378    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-run-systemd\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.233195 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232434    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-host-var-lib-cni-multus\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.233195 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232460    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea45e6d2-1a47-4956-b652-19b428800f42-registration-dir\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:25:47.233987 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232545    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjn87\" (UniqueName: \"kubernetes.io/projected/80cae79e-b60f-4efe-bb58-e77753a457e5-kube-api-access-kjn87\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.233987 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232570    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/88d9611e-c7ed-490b-b401-530d1649c873-original-pull-secret\") pod \"global-pull-secret-syncer-bqh9l\" (UID: \"88d9611e-c7ed-490b-b401-530d1649c873\") " pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:25:47.233987 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232596    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:25:47.233987 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232622    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.233987 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232647    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/80cae79e-b60f-4efe-bb58-e77753a457e5-ovnkube-config\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.233987 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232672    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwzfx\" (UniqueName: \"kubernetes.io/projected/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-kube-api-access-cwzfx\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.233987 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232694    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-multus-conf-dir\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.233987 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:47.232703    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:25:47.233987 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232719    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea45e6d2-1a47-4956-b652-19b428800f42-etc-selinux\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:25:47.233987 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232743    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-host\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.233987 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232767    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-lib-modules\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.233987 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:47.232778    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88d9611e-c7ed-490b-b401-530d1649c873-original-pull-secret podName:88d9611e-c7ed-490b-b401-530d1649c873 nodeName:}" failed. No retries permitted until 2026-10-30 20:25:47.732759352 +0000 UTC m=+2.303904436 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/88d9611e-c7ed-490b-b401-530d1649c873-original-pull-secret") pod "global-pull-secret-syncer-bqh9l" (UID: "88d9611e-c7ed-490b-b401-530d1649c873") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:25:47.233987 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232797    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-host\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.233987 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232800    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx2vz\" (UniqueName: \"kubernetes.io/projected/214f4f91-4d6e-46e0-ac1f-d23935cd7215-kube-api-access-nx2vz\") pod \"network-metrics-daemon-zcslg\" (UID: \"214f4f91-4d6e-46e0-ac1f-d23935cd7215\") " pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:25:47.233987 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232866    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/88178646-9467-4225-b362-578f31454922-hosts-file\") pod \"node-resolver-c8422\" (UID: \"88178646-9467-4225-b362-578f31454922\") " pod="openshift-dns/node-resolver-c8422"
Oct 30 20:25:47.233987 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232892    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88178646-9467-4225-b362-578f31454922-tmp-dir\") pod \"node-resolver-c8422\" (UID: \"88178646-9467-4225-b362-578f31454922\") " pod="openshift-dns/node-resolver-c8422"
Oct 30 20:25:47.233987 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232908    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea45e6d2-1a47-4956-b652-19b428800f42-etc-selinux\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:25:47.234716 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232915    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-cnibin\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:25:47.234716 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232946    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-cni-binary-copy\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:25:47.234716 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232956    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-cnibin\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:25:47.234716 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.232975    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-run-openvswitch\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.234716 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.233014    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-run-openvswitch\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.234716 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.233036    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-systemd\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.234716 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.233062    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-multus-cni-dir\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.234716 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.233085    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-host-var-lib-cni-bin\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.234716 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.233109    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-systemd-units\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.234716 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.233129    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea45e6d2-1a47-4956-b652-19b428800f42-sys-fs\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:25:47.234716 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.233151    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgvld\" (UniqueName: \"kubernetes.io/projected/111798c9-6959-4dcb-9a8a-db3037f7d662-kube-api-access-pgvld\") pod \"iptables-alerter-kwm7g\" (UID: \"111798c9-6959-4dcb-9a8a-db3037f7d662\") " pod="openshift-network-operator/iptables-alerter-kwm7g"
Oct 30 20:25:47.234716 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.233171    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-sys\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.234716 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.233189    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-var-lib-kubelet\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.234716 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.233211    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-host-run-netns\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.234716 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.233213    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:25:47.234716 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.233312    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/80cae79e-b60f-4efe-bb58-e77753a457e5-ovnkube-config\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.234716 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.233234    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkqqp\" (UniqueName: \"kubernetes.io/projected/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-kube-api-access-mkqqp\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.235305 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.233365    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-run-ovn\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.235305 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.233372    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-host-var-lib-cni-bin\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.235305 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.233392    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/80cae79e-b60f-4efe-bb58-e77753a457e5-env-overrides\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.235305 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.233440    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.235305 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.233457    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-systemd-units\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.235305 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.233472    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-multus-conf-dir\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.235305 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.233500    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-run-ovn\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.235305 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.233506    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea45e6d2-1a47-4956-b652-19b428800f42-sys-fs\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:25:47.235305 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.233510    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-cni-binary-copy\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:25:47.235305 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.233544    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-sys\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.235305 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.233548    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-var-lib-kubelet\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.235305 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.233612    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-sysctl-conf\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.235305 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.233581    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-host-run-netns\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.235305 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.233721    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-sysctl-conf\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.235305 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.233741    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/88178646-9467-4225-b362-578f31454922-hosts-file\") pod \"node-resolver-c8422\" (UID: \"88178646-9467-4225-b362-578f31454922\") " pod="openshift-dns/node-resolver-c8422"
Oct 30 20:25:47.235305 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.233800    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-tmp\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.235305 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.233853    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-systemd\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.235305 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.233872    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/214f4f91-4d6e-46e0-ac1f-d23935cd7215-metrics-certs\") pod \"network-metrics-daemon-zcslg\" (UID: \"214f4f91-4d6e-46e0-ac1f-d23935cd7215\") " pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:25:47.235847 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.233924    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-multus-cni-dir\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.235847 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.233966    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-system-cni-dir\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.235847 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:47.233994    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:25:47.235847 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.234030    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-cnibin\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.235847 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.234054    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-host-var-lib-kubelet\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.235847 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.234076    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-modprobe-d\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.235847 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.234119    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-sysctl-d\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.235847 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.234229    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-sysctl-d\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.235847 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.234277    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88178646-9467-4225-b362-578f31454922-tmp-dir\") pod \"node-resolver-c8422\" (UID: \"88178646-9467-4225-b362-578f31454922\") " pod="openshift-dns/node-resolver-c8422"
Oct 30 20:25:47.235847 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.234281    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:25:47.235847 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.234291    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-modprobe-d\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.235847 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.234328    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-system-cni-dir\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.235847 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.234360    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-cnibin\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.235847 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.234405    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-host-var-lib-kubelet\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.235847 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:47.234464    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/214f4f91-4d6e-46e0-ac1f-d23935cd7215-metrics-certs podName:214f4f91-4d6e-46e0-ac1f-d23935cd7215 nodeName:}" failed. No retries permitted until 2026-10-30 20:25:47.734449651 +0000 UTC m=+2.305594724 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/214f4f91-4d6e-46e0-ac1f-d23935cd7215-metrics-certs") pod "network-metrics-daemon-zcslg" (UID: "214f4f91-4d6e-46e0-ac1f-d23935cd7215") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:25:47.235847 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.235032    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea45e6d2-1a47-4956-b652-19b428800f42-socket-dir\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:25:47.235847 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.235075    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-tuned\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.235847 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.235127    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-cni-binary-copy\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.236343 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.235267    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/80cae79e-b60f-4efe-bb58-e77753a457e5-env-overrides\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.236343 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.235360    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/80cae79e-b60f-4efe-bb58-e77753a457e5-ovn-node-metrics-cert\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.236343 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.235461    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/89ed3d9f-e057-40a3-add4-d6374380be1f-agent-certs\") pod \"konnectivity-agent-4khkm\" (UID: \"89ed3d9f-e057-40a3-add4-d6374380be1f\") " pod="kube-system/konnectivity-agent-4khkm"
Oct 30 20:25:47.236343 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.236110    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-tmp\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.287902 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:47.287869    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:25:47.287902 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:47.287904    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:25:47.288121 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:47.287921    2575 projected.go:196] Error preparing data for projected volume kube-api-access-bb792 for pod openshift-network-diagnostics/network-check-target-pfvnn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:25:47.288121 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:47.287997    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9732ee5-c72e-44c6-9917-49cc9f81ffe0-kube-api-access-bb792 podName:c9732ee5-c72e-44c6-9917-49cc9f81ffe0 nodeName:}" failed. No retries permitted until 2026-10-30 20:25:47.787975521 +0000 UTC m=+2.359120582 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bb792" (UniqueName: "kubernetes.io/projected/c9732ee5-c72e-44c6-9917-49cc9f81ffe0-kube-api-access-bb792") pod "network-check-target-pfvnn" (UID: "c9732ee5-c72e-44c6-9917-49cc9f81ffe0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:25:47.289489 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.289459    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5zcp\" (UniqueName: \"kubernetes.io/projected/ea45e6d2-1a47-4956-b652-19b428800f42-kube-api-access-h5zcp\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:25:47.290691 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.290627    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llcbk\" (UniqueName: \"kubernetes.io/projected/8b7865f8-cfd2-4938-8621-92b63faee7af-kube-api-access-llcbk\") pod \"node-ca-rdqrc\" (UID: \"8b7865f8-cfd2-4938-8621-92b63faee7af\") " pod="openshift-image-registry/node-ca-rdqrc"
Oct 30 20:25:47.290855 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.290751    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w5vd\" (UniqueName: \"kubernetes.io/projected/88178646-9467-4225-b362-578f31454922-kube-api-access-2w5vd\") pod \"node-resolver-c8422\" (UID: \"88178646-9467-4225-b362-578f31454922\") " pod="openshift-dns/node-resolver-c8422"
Oct 30 20:25:47.292022 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.292001    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjn87\" (UniqueName: \"kubernetes.io/projected/80cae79e-b60f-4efe-bb58-e77753a457e5-kube-api-access-kjn87\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.292109 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.292094    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkqqp\" (UniqueName: \"kubernetes.io/projected/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-kube-api-access-mkqqp\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.292394 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.292374    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwzfx\" (UniqueName: \"kubernetes.io/projected/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-kube-api-access-cwzfx\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.292461 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.292448    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgvld\" (UniqueName: \"kubernetes.io/projected/111798c9-6959-4dcb-9a8a-db3037f7d662-kube-api-access-pgvld\") pod \"iptables-alerter-kwm7g\" (UID: \"111798c9-6959-4dcb-9a8a-db3037f7d662\") " pod="openshift-network-operator/iptables-alerter-kwm7g"
Oct 30 20:25:47.292791 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.292776    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4vlt\" (UniqueName: \"kubernetes.io/projected/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-kube-api-access-l4vlt\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:25:47.293597 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.293578    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx2vz\" (UniqueName: \"kubernetes.io/projected/214f4f91-4d6e-46e0-ac1f-d23935cd7215-kube-api-access-nx2vz\") pod \"network-metrics-daemon-zcslg\" (UID: \"214f4f91-4d6e-46e0-ac1f-d23935cd7215\") " pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:25:47.408598 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.408564    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:25:47.420642 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.420615    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2b77p"
Oct 30 20:25:47.451937 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.451903    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:25:47.456972 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.456953    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:25:47.457063 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.457022    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rdqrc"
Oct 30 20:25:47.480261 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.480236    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kwm7g"
Oct 30 20:25:47.485858 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.485843    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:25:47.500902 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.500883    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c8422"
Oct 30 20:25:47.505935 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.505921    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4khkm"
Oct 30 20:25:47.535484 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:47.535444    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88178646_9467_4225_b362_578f31454922.slice/crio-d70c75e394efe4b1b2216393091ec3e7cf21149ceae322a6e715b298e713dbac WatchSource:0}: Error finding container d70c75e394efe4b1b2216393091ec3e7cf21149ceae322a6e715b298e713dbac: Status 404 returned error can't find the container with id d70c75e394efe4b1b2216393091ec3e7cf21149ceae322a6e715b298e713dbac
Oct 30 20:25:47.535972 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:47.535947    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89ed3d9f_e057_40a3_add4_d6374380be1f.slice/crio-893a77ea05e781f02d98656c4e2afe76248c31558482172f870102e7960d2962 WatchSource:0}: Error finding container 893a77ea05e781f02d98656c4e2afe76248c31558482172f870102e7960d2962: Status 404 returned error can't find the container with id 893a77ea05e781f02d98656c4e2afe76248c31558482172f870102e7960d2962
Oct 30 20:25:47.537060 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:47.537039    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc591170baa3135b88257a110e786e9de.slice/crio-d61fd4b72b62bcedeb1810130637db8197fa3ad4e426a795f91c9b8f2327d1e1 WatchSource:0}: Error finding container d61fd4b72b62bcedeb1810130637db8197fa3ad4e426a795f91c9b8f2327d1e1: Status 404 returned error can't find the container with id d61fd4b72b62bcedeb1810130637db8197fa3ad4e426a795f91c9b8f2327d1e1
Oct 30 20:25:47.537479 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:47.537465    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d9f66ca5c88277c8c76e3090f63988d.slice/crio-a8ab4ef46c9d5296876a382ea4c75b1db6d6d7930fe0ae3587397615ebf918f4 WatchSource:0}: Error finding container a8ab4ef46c9d5296876a382ea4c75b1db6d6d7930fe0ae3587397615ebf918f4: Status 404 returned error can't find the container with id a8ab4ef46c9d5296876a382ea4c75b1db6d6d7930fe0ae3587397615ebf918f4
Oct 30 20:25:47.540183 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.540168    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:25:47.735940 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.735897    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/214f4f91-4d6e-46e0-ac1f-d23935cd7215-metrics-certs\") pod \"network-metrics-daemon-zcslg\" (UID: \"214f4f91-4d6e-46e0-ac1f-d23935cd7215\") " pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:25:47.736140 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.735982    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/88d9611e-c7ed-490b-b401-530d1649c873-original-pull-secret\") pod \"global-pull-secret-syncer-bqh9l\" (UID: \"88d9611e-c7ed-490b-b401-530d1649c873\") " pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:25:47.736140 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:47.736056    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:25:47.736140 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:47.736086    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:25:47.736140 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:47.736116    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/214f4f91-4d6e-46e0-ac1f-d23935cd7215-metrics-certs podName:214f4f91-4d6e-46e0-ac1f-d23935cd7215 nodeName:}" failed. No retries permitted until 2026-10-30 20:25:48.736102284 +0000 UTC m=+3.307247360 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/214f4f91-4d6e-46e0-ac1f-d23935cd7215-metrics-certs") pod "network-metrics-daemon-zcslg" (UID: "214f4f91-4d6e-46e0-ac1f-d23935cd7215") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:25:47.736140 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:47.736143    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88d9611e-c7ed-490b-b401-530d1649c873-original-pull-secret podName:88d9611e-c7ed-490b-b401-530d1649c873 nodeName:}" failed. No retries permitted until 2026-10-30 20:25:48.736130582 +0000 UTC m=+3.307275641 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/88d9611e-c7ed-490b-b401-530d1649c873-original-pull-secret") pod "global-pull-secret-syncer-bqh9l" (UID: "88d9611e-c7ed-490b-b401-530d1649c873") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:25:47.836533 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.836453    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb792\" (UniqueName: \"kubernetes.io/projected/c9732ee5-c72e-44c6-9917-49cc9f81ffe0-kube-api-access-bb792\") pod \"network-check-target-pfvnn\" (UID: \"c9732ee5-c72e-44c6-9917-49cc9f81ffe0\") " pod="openshift-network-diagnostics/network-check-target-pfvnn"
Oct 30 20:25:47.837156 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:47.836593    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:25:47.837156 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:47.836606    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:25:47.837156 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:47.836616    2575 projected.go:196] Error preparing data for projected volume kube-api-access-bb792 for pod openshift-network-diagnostics/network-check-target-pfvnn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:25:47.837156 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:47.836671    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9732ee5-c72e-44c6-9917-49cc9f81ffe0-kube-api-access-bb792 podName:c9732ee5-c72e-44c6-9917-49cc9f81ffe0 nodeName:}" failed. No retries permitted until 2026-10-30 20:25:48.836658741 +0000 UTC m=+3.407803799 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bb792" (UniqueName: "kubernetes.io/projected/c9732ee5-c72e-44c6-9917-49cc9f81ffe0-kube-api-access-bb792") pod "network-check-target-pfvnn" (UID: "c9732ee5-c72e-44c6-9917-49cc9f81ffe0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:25:47.958424 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.958348    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4khkm" event={"ID":"89ed3d9f-e057-40a3-add4-d6374380be1f","Type":"ContainerStarted","Data":"893a77ea05e781f02d98656c4e2afe76248c31558482172f870102e7960d2962"}
Oct 30 20:25:47.959531 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.959496    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c8422" event={"ID":"88178646-9467-4225-b362-578f31454922","Type":"ContainerStarted","Data":"d70c75e394efe4b1b2216393091ec3e7cf21149ceae322a6e715b298e713dbac"}
Oct 30 20:25:47.960603 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.960583    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-149.us-west-1.compute.internal" event={"ID":"9d9f66ca5c88277c8c76e3090f63988d","Type":"ContainerStarted","Data":"a8ab4ef46c9d5296876a382ea4c75b1db6d6d7930fe0ae3587397615ebf918f4"}
Oct 30 20:25:47.961661 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:47.961639    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal" event={"ID":"c591170baa3135b88257a110e786e9de","Type":"ContainerStarted","Data":"d61fd4b72b62bcedeb1810130637db8197fa3ad4e426a795f91c9b8f2327d1e1"}
Oct 30 20:25:48.110330 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:48.110257    2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:25:48.112533 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:48.112493    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:20:47 +0000 UTC" deadline="2028-05-18 10:17:05.482408951 +0000 UTC"
Oct 30 20:25:48.112617 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:48.112534    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13573h51m17.369878509s"
Oct 30 20:25:48.340172 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:48.340138    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea45e6d2_1a47_4956_b652_19b428800f42.slice/crio-0c839da3408e96750313810ca2107e90b21c719237b93eddfac858956709cad8 WatchSource:0}: Error finding container 0c839da3408e96750313810ca2107e90b21c719237b93eddfac858956709cad8: Status 404 returned error can't find the container with id 0c839da3408e96750313810ca2107e90b21c719237b93eddfac858956709cad8
Oct 30 20:25:48.363834 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:48.363812    2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:25:48.366344 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:48.366314    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80cae79e_b60f_4efe_bb58_e77753a457e5.slice/crio-e42867f37947e2331c4b1764ead6fca0dee5f574a15c6803fa80a1d6fb71f59a WatchSource:0}: Error finding container e42867f37947e2331c4b1764ead6fca0dee5f574a15c6803fa80a1d6fb71f59a: Status 404 returned error can't find the container with id e42867f37947e2331c4b1764ead6fca0dee5f574a15c6803fa80a1d6fb71f59a
Oct 30 20:25:48.384053 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:48.384025    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b7865f8_cfd2_4938_8621_92b63faee7af.slice/crio-9f5546fcc9c0952128d607267dd092c06ca09b46cec6046bab3a3c738e3040d4 WatchSource:0}: Error finding container 9f5546fcc9c0952128d607267dd092c06ca09b46cec6046bab3a3c738e3040d4: Status 404 returned error can't find the container with id 9f5546fcc9c0952128d607267dd092c06ca09b46cec6046bab3a3c738e3040d4
Oct 30 20:25:48.456556 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:25:48.456530    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7187a44e_c9e7_4d2d_be9c_f99a11c1401b.slice/crio-a06d3f2bc90ae7f931b0f96b8fca0718b523c0a8e7cb9e6fea5f6a9201479d9a WatchSource:0}: Error finding container a06d3f2bc90ae7f931b0f96b8fca0718b523c0a8e7cb9e6fea5f6a9201479d9a: Status 404 returned error can't find the container with id a06d3f2bc90ae7f931b0f96b8fca0718b523c0a8e7cb9e6fea5f6a9201479d9a
Oct 30 20:25:48.743238 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:48.743090    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/214f4f91-4d6e-46e0-ac1f-d23935cd7215-metrics-certs\") pod \"network-metrics-daemon-zcslg\" (UID: \"214f4f91-4d6e-46e0-ac1f-d23935cd7215\") " pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:25:48.743410 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:48.743279    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/88d9611e-c7ed-490b-b401-530d1649c873-original-pull-secret\") pod \"global-pull-secret-syncer-bqh9l\" (UID: \"88d9611e-c7ed-490b-b401-530d1649c873\") " pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:25:48.743483 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:48.743414    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:25:48.743543 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:48.743482    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88d9611e-c7ed-490b-b401-530d1649c873-original-pull-secret podName:88d9611e-c7ed-490b-b401-530d1649c873 nodeName:}" failed. No retries permitted until 2026-10-30 20:25:50.74346021 +0000 UTC m=+5.314605288 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/88d9611e-c7ed-490b-b401-530d1649c873-original-pull-secret") pod "global-pull-secret-syncer-bqh9l" (UID: "88d9611e-c7ed-490b-b401-530d1649c873") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:25:48.743594 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:48.743547    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:25:48.743594 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:48.743585    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/214f4f91-4d6e-46e0-ac1f-d23935cd7215-metrics-certs podName:214f4f91-4d6e-46e0-ac1f-d23935cd7215 nodeName:}" failed. No retries permitted until 2026-10-30 20:25:50.743572138 +0000 UTC m=+5.314717211 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/214f4f91-4d6e-46e0-ac1f-d23935cd7215-metrics-certs") pod "network-metrics-daemon-zcslg" (UID: "214f4f91-4d6e-46e0-ac1f-d23935cd7215") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:25:48.844713 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:48.844674    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb792\" (UniqueName: \"kubernetes.io/projected/c9732ee5-c72e-44c6-9917-49cc9f81ffe0-kube-api-access-bb792\") pod \"network-check-target-pfvnn\" (UID: \"c9732ee5-c72e-44c6-9917-49cc9f81ffe0\") " pod="openshift-network-diagnostics/network-check-target-pfvnn"
Oct 30 20:25:48.845321 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:48.845035    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:25:48.845321 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:48.845057    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:25:48.845321 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:48.845071    2575 projected.go:196] Error preparing data for projected volume kube-api-access-bb792 for pod openshift-network-diagnostics/network-check-target-pfvnn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:25:48.845321 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:48.845140    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9732ee5-c72e-44c6-9917-49cc9f81ffe0-kube-api-access-bb792 podName:c9732ee5-c72e-44c6-9917-49cc9f81ffe0 nodeName:}" failed. No retries permitted until 2026-10-30 20:25:50.845119947 +0000 UTC m=+5.416265010 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bb792" (UniqueName: "kubernetes.io/projected/c9732ee5-c72e-44c6-9917-49cc9f81ffe0-kube-api-access-bb792") pod "network-check-target-pfvnn" (UID: "c9732ee5-c72e-44c6-9917-49cc9f81ffe0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:25:48.955623 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:48.955592    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfvnn"
Oct 30 20:25:48.955810 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:48.955710    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfvnn" podUID="c9732ee5-c72e-44c6-9917-49cc9f81ffe0"
Oct 30 20:25:48.955810 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:48.955780    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:25:48.955930 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:48.955839    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bqh9l" podUID="88d9611e-c7ed-490b-b401-530d1649c873"
Oct 30 20:25:48.955930 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:48.955912    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:25:48.956008 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:48.955966    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcslg" podUID="214f4f91-4d6e-46e0-ac1f-d23935cd7215"
Oct 30 20:25:48.965625 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:48.965589    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qxwlb" event={"ID":"7187a44e-c9e7-4d2d-be9c-f99a11c1401b","Type":"ContainerStarted","Data":"a06d3f2bc90ae7f931b0f96b8fca0718b523c0a8e7cb9e6fea5f6a9201479d9a"}
Oct 30 20:25:48.967769 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:48.967738    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-59qcr" event={"ID":"d9d686e0-6edc-4db0-ab0d-4fd855c2e332","Type":"ContainerStarted","Data":"2a936c2ea9ec097a56acfe35736b879ccbc1d8d887ae734b447ad9cd1130cd51"}
Oct 30 20:25:48.969156 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:48.969131    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2b77p" event={"ID":"d83a8ce1-0cdc-410c-87fb-bfb640dfe702","Type":"ContainerStarted","Data":"1a20f9bbd988db5a16fa334184144ae38c28e82d8703967417cc8b100e7b2031"}
Oct 30 20:25:48.971801 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:48.971772    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h" event={"ID":"80cae79e-b60f-4efe-bb58-e77753a457e5","Type":"ContainerStarted","Data":"e42867f37947e2331c4b1764ead6fca0dee5f574a15c6803fa80a1d6fb71f59a"}
Oct 30 20:25:48.974922 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:48.974882    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95" event={"ID":"ea45e6d2-1a47-4956-b652-19b428800f42","Type":"ContainerStarted","Data":"0c839da3408e96750313810ca2107e90b21c719237b93eddfac858956709cad8"}
Oct 30 20:25:48.978044 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:48.978015    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kwm7g" event={"ID":"111798c9-6959-4dcb-9a8a-db3037f7d662","Type":"ContainerStarted","Data":"ffaef20d91077fb6da2ed6f19bd456f0210619e5689b483e6f58bf2166c4f1e4"}
Oct 30 20:25:48.979679 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:48.979652    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rdqrc" event={"ID":"8b7865f8-cfd2-4938-8621-92b63faee7af","Type":"ContainerStarted","Data":"9f5546fcc9c0952128d607267dd092c06ca09b46cec6046bab3a3c738e3040d4"}
Oct 30 20:25:49.113762 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:49.113717    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:20:47 +0000 UTC" deadline="2028-06-13 14:37:59.590647591 +0000 UTC"
Oct 30 20:25:49.113762 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:49.113759    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14202h12m10.476893905s"
Oct 30 20:25:50.760919 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:50.760878    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/88d9611e-c7ed-490b-b401-530d1649c873-original-pull-secret\") pod \"global-pull-secret-syncer-bqh9l\" (UID: \"88d9611e-c7ed-490b-b401-530d1649c873\") " pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:25:50.761436 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:50.760950    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/214f4f91-4d6e-46e0-ac1f-d23935cd7215-metrics-certs\") pod \"network-metrics-daemon-zcslg\" (UID: \"214f4f91-4d6e-46e0-ac1f-d23935cd7215\") " pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:25:50.761436 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:50.761097    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:25:50.761436 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:50.761159    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/214f4f91-4d6e-46e0-ac1f-d23935cd7215-metrics-certs podName:214f4f91-4d6e-46e0-ac1f-d23935cd7215 nodeName:}" failed. No retries permitted until 2026-10-30 20:25:54.761139946 +0000 UTC m=+9.332285022 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/214f4f91-4d6e-46e0-ac1f-d23935cd7215-metrics-certs") pod "network-metrics-daemon-zcslg" (UID: "214f4f91-4d6e-46e0-ac1f-d23935cd7215") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:25:50.761626 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:50.761566    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:25:50.761626 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:50.761623    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88d9611e-c7ed-490b-b401-530d1649c873-original-pull-secret podName:88d9611e-c7ed-490b-b401-530d1649c873 nodeName:}" failed. No retries permitted until 2026-10-30 20:25:54.761607257 +0000 UTC m=+9.332752331 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/88d9611e-c7ed-490b-b401-530d1649c873-original-pull-secret") pod "global-pull-secret-syncer-bqh9l" (UID: "88d9611e-c7ed-490b-b401-530d1649c873") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:25:50.862074 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:50.862034    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb792\" (UniqueName: \"kubernetes.io/projected/c9732ee5-c72e-44c6-9917-49cc9f81ffe0-kube-api-access-bb792\") pod \"network-check-target-pfvnn\" (UID: \"c9732ee5-c72e-44c6-9917-49cc9f81ffe0\") " pod="openshift-network-diagnostics/network-check-target-pfvnn"
Oct 30 20:25:50.862381 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:50.862361    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:25:50.862470 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:50.862389    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:25:50.862470 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:50.862402    2575 projected.go:196] Error preparing data for projected volume kube-api-access-bb792 for pod openshift-network-diagnostics/network-check-target-pfvnn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:25:50.862470 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:50.862463    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9732ee5-c72e-44c6-9917-49cc9f81ffe0-kube-api-access-bb792 podName:c9732ee5-c72e-44c6-9917-49cc9f81ffe0 nodeName:}" failed. No retries permitted until 2026-10-30 20:25:54.862444781 +0000 UTC m=+9.433589855 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bb792" (UniqueName: "kubernetes.io/projected/c9732ee5-c72e-44c6-9917-49cc9f81ffe0-kube-api-access-bb792") pod "network-check-target-pfvnn" (UID: "c9732ee5-c72e-44c6-9917-49cc9f81ffe0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:25:50.954756 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:50.954707    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:25:50.954941 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:50.954862    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bqh9l" podUID="88d9611e-c7ed-490b-b401-530d1649c873"
Oct 30 20:25:50.955293 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:50.955270    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:25:50.955431 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:50.955381    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcslg" podUID="214f4f91-4d6e-46e0-ac1f-d23935cd7215"
Oct 30 20:25:50.955505 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:50.955454    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfvnn"
Oct 30 20:25:50.955565 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:50.955541    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfvnn" podUID="c9732ee5-c72e-44c6-9917-49cc9f81ffe0"
Oct 30 20:25:52.955639 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:52.955601    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:25:52.956261 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:52.955764    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcslg" podUID="214f4f91-4d6e-46e0-ac1f-d23935cd7215"
Oct 30 20:25:52.956261 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:52.956254    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfvnn"
Oct 30 20:25:52.956374 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:52.956358    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfvnn" podUID="c9732ee5-c72e-44c6-9917-49cc9f81ffe0"
Oct 30 20:25:52.956435 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:52.956415    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:25:52.956521 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:52.956499    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bqh9l" podUID="88d9611e-c7ed-490b-b401-530d1649c873"
Oct 30 20:25:53.018229 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:53.018189    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal" event={"ID":"c591170baa3135b88257a110e786e9de","Type":"ContainerDied","Data":"4874c533c2cb70d6e641c0a2aed6db15c4dfa830b376680780873e212fcd2dcf"}
Oct 30 20:25:53.019691 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:53.019666    2575 generic.go:356] "Generic (PLEG): container finished" podID="c591170baa3135b88257a110e786e9de" containerID="4874c533c2cb70d6e641c0a2aed6db15c4dfa830b376680780873e212fcd2dcf" exitCode=0
Oct 30 20:25:53.034542 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:53.033683    2575 generic.go:356] "Generic (PLEG): container finished" podID="7187a44e-c9e7-4d2d-be9c-f99a11c1401b" containerID="bc09d9b2b8c85f0029a93b2f9f90135349107527cb8a7c6ea6ca16547307926e" exitCode=0
Oct 30 20:25:53.034542 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:53.033796    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qxwlb" event={"ID":"7187a44e-c9e7-4d2d-be9c-f99a11c1401b","Type":"ContainerDied","Data":"bc09d9b2b8c85f0029a93b2f9f90135349107527cb8a7c6ea6ca16547307926e"}
Oct 30 20:25:53.038291 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:53.037833    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4khkm" event={"ID":"89ed3d9f-e057-40a3-add4-d6374380be1f","Type":"ContainerStarted","Data":"641d570eeadb38a65aa743d7e4428a92c69070985bf26b4eb728248e5ad51980"}
Oct 30 20:25:53.040863 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:53.040474    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-149.us-west-1.compute.internal" event={"ID":"9d9f66ca5c88277c8c76e3090f63988d","Type":"ContainerStarted","Data":"846c2077b807dd336e69651e6c12bdb38f2b9105a3489c36a150f8ecad8485ee"}
Oct 30 20:25:53.415459 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:53.415082    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-149.us-west-1.compute.internal" podStartSLOduration=6.415060574 podStartE2EDuration="6.415060574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:53.414404659 +0000 UTC m=+7.985549741" watchObservedRunningTime="2026-10-30 20:25:53.415060574 +0000 UTC m=+7.986205660"
Oct 30 20:25:54.053767 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:54.053733    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal" event={"ID":"c591170baa3135b88257a110e786e9de","Type":"ContainerStarted","Data":"14f4850e366661720e26c84d3938106769611c518258721e1a994af71d14fa48"}
Oct 30 20:25:54.178297 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:54.178132    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal" podStartSLOduration=7.178110174 podStartE2EDuration="7.178110174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:54.177986627 +0000 UTC m=+8.749131712" watchObservedRunningTime="2026-10-30 20:25:54.178110174 +0000 UTC m=+8.749255252"
Oct 30 20:25:54.179254 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:54.179144    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4khkm" podStartSLOduration=3.12490201 podStartE2EDuration="7.179133225s" totalImagesPullingTime="4.054231215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:25:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:53.54017556 +0000 UTC m=+8.111320643" watchObservedRunningTime="2026-10-30 20:25:54.179133225 +0000 UTC m=+8.750278310"
Oct 30 20:25:54.583054 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:54.583018    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4khkm"
Oct 30 20:25:54.583564 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:54.583526    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4khkm"
Oct 30 20:25:54.796306 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:54.796267    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/88d9611e-c7ed-490b-b401-530d1649c873-original-pull-secret\") pod \"global-pull-secret-syncer-bqh9l\" (UID: \"88d9611e-c7ed-490b-b401-530d1649c873\") " pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:25:54.796507 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:54.796332    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/214f4f91-4d6e-46e0-ac1f-d23935cd7215-metrics-certs\") pod \"network-metrics-daemon-zcslg\" (UID: \"214f4f91-4d6e-46e0-ac1f-d23935cd7215\") " pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:25:54.796507 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:54.796464    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:25:54.796635 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:54.796528    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/214f4f91-4d6e-46e0-ac1f-d23935cd7215-metrics-certs podName:214f4f91-4d6e-46e0-ac1f-d23935cd7215 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:02.796510142 +0000 UTC m=+17.367655206 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/214f4f91-4d6e-46e0-ac1f-d23935cd7215-metrics-certs") pod "network-metrics-daemon-zcslg" (UID: "214f4f91-4d6e-46e0-ac1f-d23935cd7215") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:25:54.797111 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:54.797086    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:25:54.797210 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:54.797165    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88d9611e-c7ed-490b-b401-530d1649c873-original-pull-secret podName:88d9611e-c7ed-490b-b401-530d1649c873 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:02.797147546 +0000 UTC m=+17.368292611 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/88d9611e-c7ed-490b-b401-530d1649c873-original-pull-secret") pod "global-pull-secret-syncer-bqh9l" (UID: "88d9611e-c7ed-490b-b401-530d1649c873") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:25:54.897114 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:54.897072    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb792\" (UniqueName: \"kubernetes.io/projected/c9732ee5-c72e-44c6-9917-49cc9f81ffe0-kube-api-access-bb792\") pod \"network-check-target-pfvnn\" (UID: \"c9732ee5-c72e-44c6-9917-49cc9f81ffe0\") " pod="openshift-network-diagnostics/network-check-target-pfvnn"
Oct 30 20:25:54.897290 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:54.897261    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:25:54.897290 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:54.897282    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:25:54.897407 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:54.897296    2575 projected.go:196] Error preparing data for projected volume kube-api-access-bb792 for pod openshift-network-diagnostics/network-check-target-pfvnn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:25:54.897407 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:54.897358    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9732ee5-c72e-44c6-9917-49cc9f81ffe0-kube-api-access-bb792 podName:c9732ee5-c72e-44c6-9917-49cc9f81ffe0 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:02.897338803 +0000 UTC m=+17.468483879 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bb792" (UniqueName: "kubernetes.io/projected/c9732ee5-c72e-44c6-9917-49cc9f81ffe0-kube-api-access-bb792") pod "network-check-target-pfvnn" (UID: "c9732ee5-c72e-44c6-9917-49cc9f81ffe0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:25:54.955439 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:54.955408    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:25:54.955597 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:54.955574    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcslg" podUID="214f4f91-4d6e-46e0-ac1f-d23935cd7215"
Oct 30 20:25:54.955669 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:54.955652    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfvnn"
Oct 30 20:25:54.955750 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:54.955732    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfvnn" podUID="c9732ee5-c72e-44c6-9917-49cc9f81ffe0"
Oct 30 20:25:54.955799 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:54.955792    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:25:54.955916 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:54.955888    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bqh9l" podUID="88d9611e-c7ed-490b-b401-530d1649c873"
Oct 30 20:25:55.069122 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:55.069040    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rdqrc" event={"ID":"8b7865f8-cfd2-4938-8621-92b63faee7af","Type":"ContainerStarted","Data":"7d5b02e11c4a8f7c201429f4ae40e4e6101c9b46cdda447ccfb5d7ce662861a0"}
Oct 30 20:25:56.074168 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:56.074132    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95" event={"ID":"ea45e6d2-1a47-4956-b652-19b428800f42","Type":"ContainerStarted","Data":"5281a0bf3d94c1c79022649af2785b3514481a8f62761a6d149ba8dba87e4ce2"}
Oct 30 20:25:56.074553 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:56.074262    2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 20:25:56.955309 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:56.955206    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:25:56.955490 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:56.955350    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bqh9l" podUID="88d9611e-c7ed-490b-b401-530d1649c873"
Oct 30 20:25:56.955750 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:56.955728    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:25:56.955868 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:56.955845    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcslg" podUID="214f4f91-4d6e-46e0-ac1f-d23935cd7215"
Oct 30 20:25:56.955981 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:56.955936    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfvnn"
Oct 30 20:25:56.956047 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:56.956022    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfvnn" podUID="c9732ee5-c72e-44c6-9917-49cc9f81ffe0"
Oct 30 20:25:57.078695 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:57.078617    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c8422" event={"ID":"88178646-9467-4225-b362-578f31454922","Type":"ContainerStarted","Data":"b666be7959bdf700f538271bdefc706c2b4bc9381531295fa21e34583a87c514"}
Oct 30 20:25:57.080733 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:57.080705    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2b77p" event={"ID":"d83a8ce1-0cdc-410c-87fb-bfb640dfe702","Type":"ContainerStarted","Data":"5f109b767aeff8cd750dbc098e5e7df71f2ced48dbeca1deacd0d167a36e5f09"}
Oct 30 20:25:57.208962 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:57.208512    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rdqrc" podStartSLOduration=4.714261592 podStartE2EDuration="10.208491924s" totalImagesPullingTime="5.494230332s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:25:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:55.199965375 +0000 UTC m=+9.771110461" watchObservedRunningTime="2026-10-30 20:25:57.208491924 +0000 UTC m=+11.779636984"
Oct 30 20:25:57.209650 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:57.209606    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-c8422" podStartSLOduration=1.828656637 podStartE2EDuration="10.209593022s" totalImagesPullingTime="8.380936385s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:25:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:57.208481482 +0000 UTC m=+11.779626564" watchObservedRunningTime="2026-10-30 20:25:57.209593022 +0000 UTC m=+11.780738106"
Oct 30 20:25:57.247107 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:57.247072    2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 20:25:57.893847 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:57.893719    2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T20:25:57.247095487Z","UUID":"57dd4ada-54d0-497c-a996-c5ade8dc7606","Handler":null,"Name":"","Endpoint":""}
Oct 30 20:25:57.895795 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:57.895764    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 20:25:57.895795 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:57.895799    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 20:25:58.087112 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:58.087064    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95" event={"ID":"ea45e6d2-1a47-4956-b652-19b428800f42","Type":"ContainerStarted","Data":"fc6ee31e08939d901d775e9b3e803d5408e29e7914e6d5affd679bc0ff0dc4fa"}
Oct 30 20:25:58.090972 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:58.089276    2575 generic.go:356] "Generic (PLEG): container finished" podID="7187a44e-c9e7-4d2d-be9c-f99a11c1401b" containerID="909803419acdf3ca60b7a0e44e4095ddacfda4d2ef52f1f447c29bd748d65343" exitCode=0
Oct 30 20:25:58.090972 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:58.090350    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qxwlb" event={"ID":"7187a44e-c9e7-4d2d-be9c-f99a11c1401b","Type":"ContainerDied","Data":"909803419acdf3ca60b7a0e44e4095ddacfda4d2ef52f1f447c29bd748d65343"}
Oct 30 20:25:58.222758 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:58.222666    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-2b77p" podStartSLOduration=4.060205541 podStartE2EDuration="12.22265188s" totalImagesPullingTime="8.162446339s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:25:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:57.338856823 +0000 UTC m=+11.910001905" watchObservedRunningTime="2026-10-30 20:25:58.22265188 +0000 UTC m=+12.793796967"
Oct 30 20:25:58.955360 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:58.955280    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfvnn"
Oct 30 20:25:58.955502 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:58.955412    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfvnn" podUID="c9732ee5-c72e-44c6-9917-49cc9f81ffe0"
Oct 30 20:25:58.955502 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:58.955468    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:25:58.955578 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:58.955515    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bqh9l" podUID="88d9611e-c7ed-490b-b401-530d1649c873"
Oct 30 20:25:58.955578 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:58.955553    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:25:58.955650 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:25:58.955600    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcslg" podUID="214f4f91-4d6e-46e0-ac1f-d23935cd7215"
Oct 30 20:25:59.094338 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:59.094301    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95" event={"ID":"ea45e6d2-1a47-4956-b652-19b428800f42","Type":"ContainerStarted","Data":"5555681a924c36eb2bf31776dab9e8927858b9a70c89295efd5f86ab85260fe3"}
Oct 30 20:25:59.096731 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:59.096306    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kwm7g" event={"ID":"111798c9-6959-4dcb-9a8a-db3037f7d662","Type":"ContainerStarted","Data":"e9a4bc9ae103a878d16060597d580bdbed6b8745b013b3c3a3d078f1287e59cd"}
Oct 30 20:25:59.211270 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:59.211175    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95" podStartSLOduration=2.246958712 podStartE2EDuration="12.211162971s" totalImagesPullingTime="9.964204259s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:25:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:59.210943192 +0000 UTC m=+13.782088270" watchObservedRunningTime="2026-10-30 20:25:59.211162971 +0000 UTC m=+13.782308049"
Oct 30 20:25:59.334951 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:25:59.334892    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kwm7g" podStartSLOduration=3.603039246 podStartE2EDuration="12.33487529s" totalImagesPullingTime="8.731836044s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:25:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:59.334270967 +0000 UTC m=+13.905416052" watchObservedRunningTime="2026-10-30 20:25:59.33487529 +0000 UTC m=+13.906020365"
Oct 30 20:26:00.101613 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:00.101577    2575 generic.go:356] "Generic (PLEG): container finished" podID="7187a44e-c9e7-4d2d-be9c-f99a11c1401b" containerID="cc667eeb6d869e37e03a0b811e13b6c06507e125b5710a0e315d292a12f111c0" exitCode=0
Oct 30 20:26:00.102402 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:00.101644    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qxwlb" event={"ID":"7187a44e-c9e7-4d2d-be9c-f99a11c1401b","Type":"ContainerDied","Data":"cc667eeb6d869e37e03a0b811e13b6c06507e125b5710a0e315d292a12f111c0"}
Oct 30 20:26:00.103567 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:00.103535    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-59qcr" event={"ID":"d9d686e0-6edc-4db0-ab0d-4fd855c2e332","Type":"ContainerStarted","Data":"9ea9e7e8ade6e0cb6904b7858ff26754aded27d44f57ee3dd44af4bf26d3d425"}
Oct 30 20:26:00.472443 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:00.472376    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-59qcr" podStartSLOduration=1.896295253 podStartE2EDuration="13.472357784s" totalImagesPullingTime="11.576062531s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:25:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:00.472259919 +0000 UTC m=+15.043404999" watchObservedRunningTime="2026-10-30 20:26:00.472357784 +0000 UTC m=+15.043502864"
Oct 30 20:26:00.955573 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:00.955522    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfvnn"
Oct 30 20:26:00.955745 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:00.955607    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:26:00.955745 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:00.955709    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:26:00.955745 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:00.955724    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfvnn" podUID="c9732ee5-c72e-44c6-9917-49cc9f81ffe0"
Oct 30 20:26:00.955924 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:00.955807    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcslg" podUID="214f4f91-4d6e-46e0-ac1f-d23935cd7215"
Oct 30 20:26:00.955969 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:00.955932    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bqh9l" podUID="88d9611e-c7ed-490b-b401-530d1649c873"
Oct 30 20:26:02.110546 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:02.110316    2575 generic.go:356] "Generic (PLEG): container finished" podID="7187a44e-c9e7-4d2d-be9c-f99a11c1401b" containerID="a90145f99140af5ab86983c37e5bc80ca15cb9c0bb85b2ad56156f8bc89108e9" exitCode=0
Oct 30 20:26:02.111085 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:02.110384    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qxwlb" event={"ID":"7187a44e-c9e7-4d2d-be9c-f99a11c1401b","Type":"ContainerDied","Data":"a90145f99140af5ab86983c37e5bc80ca15cb9c0bb85b2ad56156f8bc89108e9"}
Oct 30 20:26:02.865661 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:02.865622    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/88d9611e-c7ed-490b-b401-530d1649c873-original-pull-secret\") pod \"global-pull-secret-syncer-bqh9l\" (UID: \"88d9611e-c7ed-490b-b401-530d1649c873\") " pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:26:02.865853 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:02.865674    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/214f4f91-4d6e-46e0-ac1f-d23935cd7215-metrics-certs\") pod \"network-metrics-daemon-zcslg\" (UID: \"214f4f91-4d6e-46e0-ac1f-d23935cd7215\") " pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:26:02.865853 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:02.865768    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:26:02.865961 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:02.865856    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88d9611e-c7ed-490b-b401-530d1649c873-original-pull-secret podName:88d9611e-c7ed-490b-b401-530d1649c873 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:18.865836853 +0000 UTC m=+33.436981927 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/88d9611e-c7ed-490b-b401-530d1649c873-original-pull-secret") pod "global-pull-secret-syncer-bqh9l" (UID: "88d9611e-c7ed-490b-b401-530d1649c873") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:26:02.865961 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:02.865772    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:26:02.865961 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:02.865906    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/214f4f91-4d6e-46e0-ac1f-d23935cd7215-metrics-certs podName:214f4f91-4d6e-46e0-ac1f-d23935cd7215 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:18.865889203 +0000 UTC m=+33.437034274 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/214f4f91-4d6e-46e0-ac1f-d23935cd7215-metrics-certs") pod "network-metrics-daemon-zcslg" (UID: "214f4f91-4d6e-46e0-ac1f-d23935cd7215") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:26:02.955112 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:02.955080    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfvnn"
Oct 30 20:26:02.955112 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:02.955129    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:26:02.955358 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:02.955158    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:26:02.955358 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:02.955253    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bqh9l" podUID="88d9611e-c7ed-490b-b401-530d1649c873"
Oct 30 20:26:02.955358 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:02.955322    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfvnn" podUID="c9732ee5-c72e-44c6-9917-49cc9f81ffe0"
Oct 30 20:26:02.955501 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:02.955406    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcslg" podUID="214f4f91-4d6e-46e0-ac1f-d23935cd7215"
Oct 30 20:26:02.966307 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:02.966281    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb792\" (UniqueName: \"kubernetes.io/projected/c9732ee5-c72e-44c6-9917-49cc9f81ffe0-kube-api-access-bb792\") pod \"network-check-target-pfvnn\" (UID: \"c9732ee5-c72e-44c6-9917-49cc9f81ffe0\") " pod="openshift-network-diagnostics/network-check-target-pfvnn"
Oct 30 20:26:02.966422 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:02.966411    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:26:02.966481 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:02.966427    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:26:02.966481 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:02.966449    2575 projected.go:196] Error preparing data for projected volume kube-api-access-bb792 for pod openshift-network-diagnostics/network-check-target-pfvnn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:26:02.966553 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:02.966494    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9732ee5-c72e-44c6-9917-49cc9f81ffe0-kube-api-access-bb792 podName:c9732ee5-c72e-44c6-9917-49cc9f81ffe0 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:18.96647958 +0000 UTC m=+33.537624638 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bb792" (UniqueName: "kubernetes.io/projected/c9732ee5-c72e-44c6-9917-49cc9f81ffe0-kube-api-access-bb792") pod "network-check-target-pfvnn" (UID: "c9732ee5-c72e-44c6-9917-49cc9f81ffe0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:26:04.118446 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:04.118253    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q4f4h_80cae79e-b60f-4efe-bb58-e77753a457e5/ovn-acl-logging/0.log"
Oct 30 20:26:04.118919 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:04.118894    2575 generic.go:356] "Generic (PLEG): container finished" podID="80cae79e-b60f-4efe-bb58-e77753a457e5" containerID="cb54274f3a3593acf07f66429cd4d519074103fea0a4321ba5607d5f526d82ea" exitCode=1
Oct 30 20:26:04.118971 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:04.118932    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h" event={"ID":"80cae79e-b60f-4efe-bb58-e77753a457e5","Type":"ContainerStarted","Data":"ab7f3d16b113fcced4c7bb569c66e2f7fe505b1678825f1437419ca042fb72de"}
Oct 30 20:26:04.119004 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:04.118994    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h" event={"ID":"80cae79e-b60f-4efe-bb58-e77753a457e5","Type":"ContainerStarted","Data":"3de78cf8e21d6f4244cba4107db71c3675a041e56f69e0731dffed6b15e0afe9"}
Oct 30 20:26:04.119037 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:04.119025    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h" event={"ID":"80cae79e-b60f-4efe-bb58-e77753a457e5","Type":"ContainerStarted","Data":"ddc831f281fda875275af04d81842dbb144ec77d868ad9ab739e1d225a4f107d"}
Oct 30 20:26:04.119065 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:04.119052    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h" event={"ID":"80cae79e-b60f-4efe-bb58-e77753a457e5","Type":"ContainerStarted","Data":"2dca868d8dc599d25604f3c86101b3102383dcbb36d0736ba32b4b9bff5de78e"}
Oct 30 20:26:04.119092 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:04.119085    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h" event={"ID":"80cae79e-b60f-4efe-bb58-e77753a457e5","Type":"ContainerDied","Data":"cb54274f3a3593acf07f66429cd4d519074103fea0a4321ba5607d5f526d82ea"}
Oct 30 20:26:04.119126 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:04.119113    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h" event={"ID":"80cae79e-b60f-4efe-bb58-e77753a457e5","Type":"ContainerStarted","Data":"559533b4bf70035d020391ab8a2150416b635d402726d6c330d5d54b491c6889"}
Oct 30 20:26:04.955460 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:04.955422    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfvnn"
Oct 30 20:26:04.955460 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:04.955453    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:26:04.955661 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:04.955564    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfvnn" podUID="c9732ee5-c72e-44c6-9917-49cc9f81ffe0"
Oct 30 20:26:04.955661 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:04.955579    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcslg" podUID="214f4f91-4d6e-46e0-ac1f-d23935cd7215"
Oct 30 20:26:04.955661 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:04.955617    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:26:04.955772 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:04.955663    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bqh9l" podUID="88d9611e-c7ed-490b-b401-530d1649c873"
Oct 30 20:26:05.123796 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:05.123761    2575 generic.go:356] "Generic (PLEG): container finished" podID="7187a44e-c9e7-4d2d-be9c-f99a11c1401b" containerID="41ae2317108bb4d64c40d11f6c0137844f6cd4b4ac18dbdf44d473a8bf1c9eb7" exitCode=0
Oct 30 20:26:05.124286 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:05.123806    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qxwlb" event={"ID":"7187a44e-c9e7-4d2d-be9c-f99a11c1401b","Type":"ContainerDied","Data":"41ae2317108bb4d64c40d11f6c0137844f6cd4b4ac18dbdf44d473a8bf1c9eb7"}
Oct 30 20:26:06.129212 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:06.129187    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q4f4h_80cae79e-b60f-4efe-bb58-e77753a457e5/ovn-acl-logging/0.log"
Oct 30 20:26:06.129657 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:06.129631    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h" event={"ID":"80cae79e-b60f-4efe-bb58-e77753a457e5","Type":"ContainerStarted","Data":"e193f90c4c7f6025248f92808dcbac479e983c7b718e99f9a94210b914fa770f"}
Oct 30 20:26:06.133136 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:06.133109    2575 generic.go:356] "Generic (PLEG): container finished" podID="7187a44e-c9e7-4d2d-be9c-f99a11c1401b" containerID="019e1ce8429641b2ed5debe382e0949988fa1698c64bfc9f75a32d1b2a6cfbbc" exitCode=0
Oct 30 20:26:06.133259 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:06.133145    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qxwlb" event={"ID":"7187a44e-c9e7-4d2d-be9c-f99a11c1401b","Type":"ContainerDied","Data":"019e1ce8429641b2ed5debe382e0949988fa1698c64bfc9f75a32d1b2a6cfbbc"}
Oct 30 20:26:06.955121 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:06.955087    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfvnn"
Oct 30 20:26:06.955259 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:06.955153    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:26:06.955259 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:06.955242    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfvnn" podUID="c9732ee5-c72e-44c6-9917-49cc9f81ffe0"
Oct 30 20:26:06.955259 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:06.955250    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:26:06.955382 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:06.955329    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcslg" podUID="214f4f91-4d6e-46e0-ac1f-d23935cd7215"
Oct 30 20:26:06.955415 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:06.955397    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bqh9l" podUID="88d9611e-c7ed-490b-b401-530d1649c873"
Oct 30 20:26:07.138095 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:07.138061    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qxwlb" event={"ID":"7187a44e-c9e7-4d2d-be9c-f99a11c1401b","Type":"ContainerStarted","Data":"62082fadf73c7d9c7e86f9a06c092c6b9f563d44a04ce98d9d11fa6272fcd78b"}
Oct 30 20:26:07.258509 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:07.258402    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qxwlb" podStartSLOduration=8.693585973 podStartE2EDuration="21.258387858s" totalImagesPullingTime="12.564801885s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:25:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:07.258249675 +0000 UTC m=+21.829394755" watchObservedRunningTime="2026-10-30 20:26:07.258387858 +0000 UTC m=+21.829532936"
Oct 30 20:26:08.955642 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:08.955614    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfvnn"
Oct 30 20:26:08.956291 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:08.955620    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:26:08.956291 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:08.955707    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfvnn" podUID="c9732ee5-c72e-44c6-9917-49cc9f81ffe0"
Oct 30 20:26:08.956291 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:08.955620    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:26:08.956291 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:08.955796    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcslg" podUID="214f4f91-4d6e-46e0-ac1f-d23935cd7215"
Oct 30 20:26:08.956291 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:08.955923    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bqh9l" podUID="88d9611e-c7ed-490b-b401-530d1649c873"
Oct 30 20:26:09.145180 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:09.145029    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q4f4h_80cae79e-b60f-4efe-bb58-e77753a457e5/ovn-acl-logging/0.log"
Oct 30 20:26:09.145503 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:09.145483    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h" event={"ID":"80cae79e-b60f-4efe-bb58-e77753a457e5","Type":"ContainerStarted","Data":"89abd503fb12a3148ea126c6cd09173c931682db21677133d82efff322ace33b"}
Oct 30 20:26:09.145772 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:09.145757    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:09.145847 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:09.145836    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:09.145907 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:09.145863    2575 scope.go:122] "RemoveContainer" containerID="cb54274f3a3593acf07f66429cd4d519074103fea0a4321ba5607d5f526d82ea"
Oct 30 20:26:09.163551 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:09.163531    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:10.150566 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:10.150540    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q4f4h_80cae79e-b60f-4efe-bb58-e77753a457e5/ovn-acl-logging/0.log"
Oct 30 20:26:10.150977 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:10.150910    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h" event={"ID":"80cae79e-b60f-4efe-bb58-e77753a457e5","Type":"ContainerStarted","Data":"63a0b5443978cae4d69329795494ea62479c475f15ca9a1ca10d5ba5dfc58568"}
Oct 30 20:26:10.151147 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:10.151116    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:10.166444 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:10.166418    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:10.331658 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:10.331612    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h" podStartSLOduration=8.138636018 podStartE2EDuration="23.331599364s" totalImagesPullingTime="15.192963346s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:25:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:10.330696131 +0000 UTC m=+24.901841210" watchObservedRunningTime="2026-10-30 20:26:10.331599364 +0000 UTC m=+24.902744442"
Oct 30 20:26:10.955048 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:10.955019    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:26:10.955199 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:10.955076    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfvnn"
Oct 30 20:26:10.955199 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:10.955075    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:26:10.955199 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:10.955180    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfvnn" podUID="c9732ee5-c72e-44c6-9917-49cc9f81ffe0"
Oct 30 20:26:10.955307 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:10.955221    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcslg" podUID="214f4f91-4d6e-46e0-ac1f-d23935cd7215"
Oct 30 20:26:10.955307 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:10.955299    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bqh9l" podUID="88d9611e-c7ed-490b-b401-530d1649c873"
Oct 30 20:26:11.047188 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:11.047155    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bqh9l"]
Oct 30 20:26:11.050637 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:11.050593    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pfvnn"]
Oct 30 20:26:11.051417 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:11.051396    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zcslg"]
Oct 30 20:26:11.153481 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:11.153447    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:26:11.153481 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:11.153462    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfvnn"
Oct 30 20:26:11.153481 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:11.153447    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:26:11.153965 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:11.153574    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfvnn" podUID="c9732ee5-c72e-44c6-9917-49cc9f81ffe0"
Oct 30 20:26:11.153965 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:11.153620    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bqh9l" podUID="88d9611e-c7ed-490b-b401-530d1649c873"
Oct 30 20:26:11.153965 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:11.153733    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcslg" podUID="214f4f91-4d6e-46e0-ac1f-d23935cd7215"
Oct 30 20:26:12.955178 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:12.955142    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:26:12.955678 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:12.955210    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfvnn"
Oct 30 20:26:12.955678 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:12.955210    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:26:12.955678 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:12.955309    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfvnn" podUID="c9732ee5-c72e-44c6-9917-49cc9f81ffe0"
Oct 30 20:26:12.955678 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:12.955375    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bqh9l" podUID="88d9611e-c7ed-490b-b401-530d1649c873"
Oct 30 20:26:12.955678 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:12.955455    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcslg" podUID="214f4f91-4d6e-46e0-ac1f-d23935cd7215"
Oct 30 20:26:14.955114 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:14.955073    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:26:14.955114 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:14.955093    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:26:14.955569 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:14.955083    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfvnn"
Oct 30 20:26:14.955569 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:14.955178    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bqh9l" podUID="88d9611e-c7ed-490b-b401-530d1649c873"
Oct 30 20:26:14.955569 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:14.955262    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfvnn" podUID="c9732ee5-c72e-44c6-9917-49cc9f81ffe0"
Oct 30 20:26:14.955569 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:14.955349    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zcslg" podUID="214f4f91-4d6e-46e0-ac1f-d23935cd7215"
Oct 30 20:26:16.829365 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:16.829205    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-149.us-west-1.compute.internal" event="NodeReady"
Oct 30 20:26:16.829751 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:16.829459    2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 20:26:16.945515 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:16.945481    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kgsjn"]
Oct 30 20:26:16.952736 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:16.952712    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf"]
Oct 30 20:26:16.952880 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:16.952853    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-kgsjn"
Oct 30 20:26:16.959235 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:16.958917    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf"
Oct 30 20:26:16.959235 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:16.958944    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-brf75"]
Oct 30 20:26:16.959235 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:16.958942    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:26:16.959235 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:16.959050    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:26:16.959525 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:16.959509    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfvnn"
Oct 30 20:26:16.965802 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:16.965782    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-c5b2k"]
Oct 30 20:26:16.965936 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:16.965904    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-brf75"
Oct 30 20:26:16.974842 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:16.974809    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-9rs22"]
Oct 30 20:26:16.974940 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:16.974885    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-c5b2k"
Oct 30 20:26:16.983331 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:16.983304    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-666dd86ff-dnnq5"]
Oct 30 20:26:16.983450 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:16.983432    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9rs22"
Oct 30 20:26:16.989232 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:16.989208    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:16.989790 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:16.989091    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-m9n59"]
Oct 30 20:26:16.995235 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:16.995217    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-968d68cdf-vwtgc"]
Oct 30 20:26:16.995425 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:16.995408    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-m9n59"
Oct 30 20:26:17.001323 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.001304    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-l8qvb"]
Oct 30 20:26:17.001572 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.001549    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:17.010720 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.010705    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-tp7sc"]
Oct 30 20:26:17.010883 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.010812    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb"
Oct 30 20:26:17.015801 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.015778    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 20:26:17.015939 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.015854    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 20:26:17.016138 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.016119    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 20:26:17.016220 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.016158    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:26:17.017069 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.017053    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 20:26:17.019900 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.019869    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-4v8mv"]
Oct 30 20:26:17.020022 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.020006    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tp7sc"
Oct 30 20:26:17.027166 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.027146    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 20:26:17.027166 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.027157    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 20:26:17.027279 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.027220    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-78mnk\""
Oct 30 20:26:17.028100 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.028082    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z9tdd"]
Oct 30 20:26:17.028221 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.028205    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-4v8mv"
Oct 30 20:26:17.042224 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.042204    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-bhrtb"]
Oct 30 20:26:17.042311 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.042277    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z9tdd"
Oct 30 20:26:17.058489 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.058468    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7db8fb6d4-8n6g7"]
Oct 30 20:26:17.058578 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.058555    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bhrtb"
Oct 30 20:26:17.070568 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.070546    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f3f7114f-f896-4fc9-b98a-e47574e7720d-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-m9n59\" (UID: \"f3f7114f-f896-4fc9-b98a-e47574e7720d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-m9n59"
Oct 30 20:26:17.070691 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.070575    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/91fe9e06-0c30-4000-bddb-2ba89cd048cc-config\") pod \"console-operator-7cd56b8dbc-l8qvb\" (UID: \"91fe9e06-0c30-4000-bddb-2ba89cd048cc\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb"
Oct 30 20:26:17.070691 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.070594    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krgxb\" (UniqueName: \"kubernetes.io/projected/533fb89e-0b69-4c79-9cbf-7baa5f725219-kube-api-access-krgxb\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:17.070691 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.070618    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f3f7114f-f896-4fc9-b98a-e47574e7720d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-m9n59\" (UID: \"f3f7114f-f896-4fc9-b98a-e47574e7720d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-m9n59"
Oct 30 20:26:17.070691 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.070674    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/91fe9e06-0c30-4000-bddb-2ba89cd048cc-serving-cert\") pod \"console-operator-7cd56b8dbc-l8qvb\" (UID: \"91fe9e06-0c30-4000-bddb-2ba89cd048cc\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb"
Oct 30 20:26:17.070850 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.070720    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ce6c7b90-6b06-4c81-bf95-539b71e2aaed-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf\" (UID: \"ce6c7b90-6b06-4c81-bf95-539b71e2aaed\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf"
Oct 30 20:26:17.070850 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.070747    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-default-certificate\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:17.070850 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.070768    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ce6c7b90-6b06-4c81-bf95-539b71e2aaed-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf\" (UID: \"ce6c7b90-6b06-4c81-bf95-539b71e2aaed\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf"
Oct 30 20:26:17.070850 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.070794    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-registry-tls\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:17.071012 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.070855    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqftg\" (UniqueName: \"kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-kube-api-access-nqftg\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:17.071012 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.070903    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tpmm\" (UniqueName: \"kubernetes.io/projected/ce6c7b90-6b06-4c81-bf95-539b71e2aaed-kube-api-access-8tpmm\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf\" (UID: \"ce6c7b90-6b06-4c81-bf95-539b71e2aaed\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf"
Oct 30 20:26:17.071119 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.071073    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9f908129-4500-4433-871c-032c29a55dd9-ca-trust-extracted\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:17.071119 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.071100    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-metrics-certs\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:17.071224 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.071123    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv49q\" (UniqueName: \"kubernetes.io/projected/08b2f787-ad25-47f9-8ef8-724308458f88-kube-api-access-hv49q\") pod \"kas-connection-checker-685895c7c9-brf75\" (UID: \"08b2f787-ad25-47f9-8ef8-724308458f88\") " pod="kube-system/kas-connection-checker-685895c7c9-brf75"
Oct 30 20:26:17.071579 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.071560    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d5312b70-82df-4e75-a300-b8e1d63a75f3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9rs22\" (UID: \"d5312b70-82df-4e75-a300-b8e1d63a75f3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9rs22"
Oct 30 20:26:17.071640 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.071605    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/533fb89e-0b69-4c79-9cbf-7baa5f725219-service-ca-bundle\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:17.071640 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.071624    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-bound-sa-token\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:17.071743 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.071643    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94tlh\" (UniqueName: \"kubernetes.io/projected/91fe9e06-0c30-4000-bddb-2ba89cd048cc-kube-api-access-94tlh\") pod \"console-operator-7cd56b8dbc-l8qvb\" (UID: \"91fe9e06-0c30-4000-bddb-2ba89cd048cc\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb"
Oct 30 20:26:17.071743 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.071665    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9f908129-4500-4433-871c-032c29a55dd9-registry-certificates\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:17.071867 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.071735    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9f908129-4500-4433-871c-032c29a55dd9-image-registry-private-configuration\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:17.071867 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.071777    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8hdc\" (UniqueName: \"kubernetes.io/projected/f4913de8-6db5-413a-938f-abc50a5460e7-kube-api-access-d8hdc\") pod \"kas-connection-checker-685895c7c9-kgsjn\" (UID: \"f4913de8-6db5-413a-938f-abc50a5460e7\") " pod="kube-system/kas-connection-checker-685895c7c9-kgsjn"
Oct 30 20:26:17.071867 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.071797    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-mql5m"]
Oct 30 20:26:17.071867 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.071855    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8nqb\" (UniqueName: \"kubernetes.io/projected/78c7e317-9d58-4752-9804-92ac45b5afb7-kube-api-access-m8nqb\") pod \"kas-connection-checker-685895c7c9-c5b2k\" (UID: \"78c7e317-9d58-4752-9804-92ac45b5afb7\") " pod="kube-system/kas-connection-checker-685895c7c9-c5b2k"
Oct 30 20:26:17.072046 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.071876    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wb98\" (UniqueName: \"kubernetes.io/projected/d5312b70-82df-4e75-a300-b8e1d63a75f3-kube-api-access-5wb98\") pod \"cluster-monitoring-operator-884576c57-9rs22\" (UID: \"d5312b70-82df-4e75-a300-b8e1d63a75f3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9rs22"
Oct 30 20:26:17.072046 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.071896    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-stats-auth\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:17.072046 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.071898    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:17.072046 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.071938    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f908129-4500-4433-871c-032c29a55dd9-trusted-ca\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:17.072046 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.071997    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9f908129-4500-4433-871c-032c29a55dd9-installation-pull-secrets\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:17.072046 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.072037    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/91fe9e06-0c30-4000-bddb-2ba89cd048cc-trusted-ca\") pod \"console-operator-7cd56b8dbc-l8qvb\" (UID: \"91fe9e06-0c30-4000-bddb-2ba89cd048cc\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb"
Oct 30 20:26:17.072277 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.072057    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d5312b70-82df-4e75-a300-b8e1d63a75f3-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-9rs22\" (UID: \"d5312b70-82df-4e75-a300-b8e1d63a75f3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9rs22"
Oct 30 20:26:17.072277 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.072084    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsx89\" (UniqueName: \"kubernetes.io/projected/f3f7114f-f896-4fc9-b98a-e47574e7720d-kube-api-access-wsx89\") pod \"cluster-samples-operator-6ddb6c8b94-m9n59\" (UID: \"f3f7114f-f896-4fc9-b98a-e47574e7720d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-m9n59"
Oct 30 20:26:17.080578 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.080443    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-h2vh7"]
Oct 30 20:26:17.080578 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.080561    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-mql5m"
Oct 30 20:26:17.095134 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.094962    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mrnvn"]
Oct 30 20:26:17.095275 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.095205    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h2vh7"
Oct 30 20:26:17.105021 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.105000    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kgsjn"]
Oct 30 20:26:17.105201 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.105186    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf"]
Oct 30 20:26:17.105423 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.105409    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-c5b2k"]
Oct 30 20:26:17.105517 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.105437    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-666dd86ff-dnnq5"]
Oct 30 20:26:17.105517 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.105458    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-m9n59"]
Oct 30 20:26:17.105517 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.105483    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-968d68cdf-vwtgc"]
Oct 30 20:26:17.105517 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.105505    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-9rs22"]
Oct 30 20:26:17.105738 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.105528    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-tp7sc"]
Oct 30 20:26:17.105738 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.105550    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-l8qvb"]
Oct 30 20:26:17.105738 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.105578    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-mql5m"]
Oct 30 20:26:17.105738 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.105597    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h2vh7"]
Oct 30 20:26:17.105738 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.105621    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-brf75"]
Oct 30 20:26:17.105738 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.105645    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mrnvn"]
Oct 30 20:26:17.105738 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.105132    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mrnvn"
Oct 30 20:26:17.116554 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.116515    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-4v8mv"]
Oct 30 20:26:17.119021 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.119002    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 20:26:17.119110 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.119076    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 20:26:17.119214 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.119196    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 20:26:17.119410 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.119212    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 20:26:17.119410 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.119405    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 20:26:17.119572 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.119485    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 20:26:17.120245 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.120047    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:26:17.120245 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.120086    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 20:26:17.135480 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.135390    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:26:17.135480 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.135392    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 20:26:17.135480 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.135459    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:26:17.135480 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.135459    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 20:26:17.135759 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.135466    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 20:26:17.137834 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.137799    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 20:26:17.141718 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.141698    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 20:26:17.145725 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.145699    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:26:17.145725 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.145720    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 20:26:17.145892 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.145757    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 20:26:17.149322 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.149304    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 20:26:17.162559 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.162538    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4khkm"
Oct 30 20:26:17.162677 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.162664    2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 20:26:17.163125 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.163110    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4khkm"
Oct 30 20:26:17.169207 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.169187    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 20:26:17.173343 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.173322    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgf7l\" (UniqueName: \"kubernetes.io/projected/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-kube-api-access-zgf7l\") pod \"dns-default-h2vh7\" (UID: \"4c7e75b7-c3cd-4791-9d39-9120f6fd95dc\") " pod="openshift-dns/dns-default-h2vh7"
Oct 30 20:26:17.173425 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.173353    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-bound-sa-token\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:17.173425 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.173385    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9f908129-4500-4433-871c-032c29a55dd9-image-registry-private-configuration\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:17.173537 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.173441    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8hdc\" (UniqueName: \"kubernetes.io/projected/f4913de8-6db5-413a-938f-abc50a5460e7-kube-api-access-d8hdc\") pod \"kas-connection-checker-685895c7c9-kgsjn\" (UID: \"f4913de8-6db5-413a-938f-abc50a5460e7\") " pod="kube-system/kas-connection-checker-685895c7c9-kgsjn"
Oct 30 20:26:17.173537 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.173481    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/05f22644-0884-434d-b56b-193d9554b505-registry-certificates\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:17.173633 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.173537    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/05f22644-0884-434d-b56b-193d9554b505-installation-pull-secrets\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:17.173633 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.173587    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8nqb\" (UniqueName: \"kubernetes.io/projected/78c7e317-9d58-4752-9804-92ac45b5afb7-kube-api-access-m8nqb\") pod \"kas-connection-checker-685895c7c9-c5b2k\" (UID: \"78c7e317-9d58-4752-9804-92ac45b5afb7\") " pod="kube-system/kas-connection-checker-685895c7c9-c5b2k"
Oct 30 20:26:17.173633 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.173615    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-tmp-dir\") pod \"dns-default-h2vh7\" (UID: \"4c7e75b7-c3cd-4791-9d39-9120f6fd95dc\") " pod="openshift-dns/dns-default-h2vh7"
Oct 30 20:26:17.173798 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.173643    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf-serving-cert\") pod \"insights-operator-7c4986d449-mql5m\" (UID: \"8840f0ca-269e-4924-8b3d-5fe6ff4e90cf\") " pod="openshift-insights/insights-operator-7c4986d449-mql5m"
Oct 30 20:26:17.173798 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.173672    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wb98\" (UniqueName: \"kubernetes.io/projected/d5312b70-82df-4e75-a300-b8e1d63a75f3-kube-api-access-5wb98\") pod \"cluster-monitoring-operator-884576c57-9rs22\" (UID: \"d5312b70-82df-4e75-a300-b8e1d63a75f3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9rs22"
Oct 30 20:26:17.173798 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.173709    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-stats-auth\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:17.173798 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.173733    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/70fa9600-c497-4deb-a133-3cb24f1c8050-nginx-conf\") pod \"networking-console-plugin-86c478bb88-4v8mv\" (UID: \"70fa9600-c497-4deb-a133-3cb24f1c8050\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4v8mv"
Oct 30 20:26:17.173798 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.173780    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsmc8\" (UniqueName: \"kubernetes.io/projected/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf-kube-api-access-tsmc8\") pod \"insights-operator-7c4986d449-mql5m\" (UID: \"8840f0ca-269e-4924-8b3d-5fe6ff4e90cf\") " pod="openshift-insights/insights-operator-7c4986d449-mql5m"
Oct 30 20:26:17.173991 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.173842    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f908129-4500-4433-871c-032c29a55dd9-trusted-ca\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:17.173991 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.173879    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9f908129-4500-4433-871c-032c29a55dd9-installation-pull-secrets\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:17.173991 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.173920    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-metrics-tls\") pod \"dns-default-h2vh7\" (UID: \"4c7e75b7-c3cd-4791-9d39-9120f6fd95dc\") " pod="openshift-dns/dns-default-h2vh7"
Oct 30 20:26:17.173991 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.173957    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/91fe9e06-0c30-4000-bddb-2ba89cd048cc-trusted-ca\") pod \"console-operator-7cd56b8dbc-l8qvb\" (UID: \"91fe9e06-0c30-4000-bddb-2ba89cd048cc\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb"
Oct 30 20:26:17.173991 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.173975    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d5312b70-82df-4e75-a300-b8e1d63a75f3-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-9rs22\" (UID: \"d5312b70-82df-4e75-a300-b8e1d63a75f3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9rs22"
Oct 30 20:26:17.173991 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.173990    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsx89\" (UniqueName: \"kubernetes.io/projected/f3f7114f-f896-4fc9-b98a-e47574e7720d-kube-api-access-wsx89\") pod \"cluster-samples-operator-6ddb6c8b94-m9n59\" (UID: \"f3f7114f-f896-4fc9-b98a-e47574e7720d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-m9n59"
Oct 30 20:26:17.174202 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.174008    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-config-volume\") pod \"dns-default-h2vh7\" (UID: \"4c7e75b7-c3cd-4791-9d39-9120f6fd95dc\") " pod="openshift-dns/dns-default-h2vh7"
Oct 30 20:26:17.174202 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.174031    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f3f7114f-f896-4fc9-b98a-e47574e7720d-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-m9n59\" (UID: \"f3f7114f-f896-4fc9-b98a-e47574e7720d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-m9n59"
Oct 30 20:26:17.174202 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.174047    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70fa9600-c497-4deb-a133-3cb24f1c8050-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4v8mv\" (UID: \"70fa9600-c497-4deb-a133-3cb24f1c8050\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4v8mv"
Oct 30 20:26:17.174202 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.174065    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf-tmp\") pod \"insights-operator-7c4986d449-mql5m\" (UID: \"8840f0ca-269e-4924-8b3d-5fe6ff4e90cf\") " pod="openshift-insights/insights-operator-7c4986d449-mql5m"
Oct 30 20:26:17.174202 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.174087    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-mql5m\" (UID: \"8840f0ca-269e-4924-8b3d-5fe6ff4e90cf\") " pod="openshift-insights/insights-operator-7c4986d449-mql5m"
Oct 30 20:26:17.174202 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.174178    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/91fe9e06-0c30-4000-bddb-2ba89cd048cc-config\") pod \"console-operator-7cd56b8dbc-l8qvb\" (UID: \"91fe9e06-0c30-4000-bddb-2ba89cd048cc\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb"
Oct 30 20:26:17.174473 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.174241    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64rvp\" (UniqueName: \"kubernetes.io/projected/5040db9b-91d3-4d89-adaa-d1394cc3a617-kube-api-access-64rvp\") pod \"service-ca-operator-544d749c8c-tp7sc\" (UID: \"5040db9b-91d3-4d89-adaa-d1394cc3a617\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tp7sc"
Oct 30 20:26:17.174473 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.174291    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krgxb\" (UniqueName: \"kubernetes.io/projected/533fb89e-0b69-4c79-9cbf-7baa5f725219-kube-api-access-krgxb\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:17.174473 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.174307    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f3f7114f-f896-4fc9-b98a-e47574e7720d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-m9n59\" (UID: \"f3f7114f-f896-4fc9-b98a-e47574e7720d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-m9n59"
Oct 30 20:26:17.174473 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.174327    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/91fe9e06-0c30-4000-bddb-2ba89cd048cc-serving-cert\") pod \"console-operator-7cd56b8dbc-l8qvb\" (UID: \"91fe9e06-0c30-4000-bddb-2ba89cd048cc\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb"
Oct 30 20:26:17.174473 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.174354    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ce6c7b90-6b06-4c81-bf95-539b71e2aaed-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf\" (UID: \"ce6c7b90-6b06-4c81-bf95-539b71e2aaed\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf"
Oct 30 20:26:17.174473 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.174381    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1e8d4f3-e4a1-416c-bbc7-2c76063bfc24-cert\") pod \"ingress-canary-mrnvn\" (UID: \"c1e8d4f3-e4a1-416c-bbc7-2c76063bfc24\") " pod="openshift-ingress-canary/ingress-canary-mrnvn"
Oct 30 20:26:17.174473 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.174410    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf-service-ca-bundle\") pod \"insights-operator-7c4986d449-mql5m\" (UID: \"8840f0ca-269e-4924-8b3d-5fe6ff4e90cf\") " pod="openshift-insights/insights-operator-7c4986d449-mql5m"
Oct 30 20:26:17.174920 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.174547    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-default-certificate\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:17.174920 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.174582    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ce6c7b90-6b06-4c81-bf95-539b71e2aaed-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf\" (UID: \"ce6c7b90-6b06-4c81-bf95-539b71e2aaed\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf"
Oct 30 20:26:17.174920 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.174615    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-registry-tls\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:17.174920 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.174641    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-registry-tls\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:17.174920 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.174663    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fxk9\" (UniqueName: \"kubernetes.io/projected/c1e8d4f3-e4a1-416c-bbc7-2c76063bfc24-kube-api-access-7fxk9\") pod \"ingress-canary-mrnvn\" (UID: \"c1e8d4f3-e4a1-416c-bbc7-2c76063bfc24\") " pod="openshift-ingress-canary/ingress-canary-mrnvn"
Oct 30 20:26:17.174920 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.174690    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f3f7114f-f896-4fc9-b98a-e47574e7720d-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-m9n59\" (UID: \"f3f7114f-f896-4fc9-b98a-e47574e7720d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-m9n59"
Oct 30 20:26:17.174920 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.174729    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqftg\" (UniqueName: \"kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-kube-api-access-nqftg\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:17.174920 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.174739    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d5312b70-82df-4e75-a300-b8e1d63a75f3-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-9rs22\" (UID: \"d5312b70-82df-4e75-a300-b8e1d63a75f3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9rs22"
Oct 30 20:26:17.174920 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.174775    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5040db9b-91d3-4d89-adaa-d1394cc3a617-config\") pod \"service-ca-operator-544d749c8c-tp7sc\" (UID: \"5040db9b-91d3-4d89-adaa-d1394cc3a617\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tp7sc"
Oct 30 20:26:17.174920 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.174803    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/05f22644-0884-434d-b56b-193d9554b505-ca-trust-extracted\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:17.174920 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.174848    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/05f22644-0884-434d-b56b-193d9554b505-trusted-ca\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:17.174920 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.174898    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tpmm\" (UniqueName: \"kubernetes.io/projected/ce6c7b90-6b06-4c81-bf95-539b71e2aaed-kube-api-access-8tpmm\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf\" (UID: \"ce6c7b90-6b06-4c81-bf95-539b71e2aaed\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf"
Oct 30 20:26:17.175433 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.174974    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9f908129-4500-4433-871c-032c29a55dd9-ca-trust-extracted\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:17.175433 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.174993    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dx86\" (UniqueName: \"kubernetes.io/projected/a0790a28-d605-4c0c-a652-41ac6ef22e09-kube-api-access-4dx86\") pod \"volume-data-source-validator-58c9f5f46d-z9tdd\" (UID: \"a0790a28-d605-4c0c-a652-41ac6ef22e09\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z9tdd"
Oct 30 20:26:17.175433 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.175013    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-metrics-certs\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:17.175433 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.175010    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/91fe9e06-0c30-4000-bddb-2ba89cd048cc-config\") pod \"console-operator-7cd56b8dbc-l8qvb\" (UID: \"91fe9e06-0c30-4000-bddb-2ba89cd048cc\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb"
Oct 30 20:26:17.175433 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.175028    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5040db9b-91d3-4d89-adaa-d1394cc3a617-serving-cert\") pod \"service-ca-operator-544d749c8c-tp7sc\" (UID: \"5040db9b-91d3-4d89-adaa-d1394cc3a617\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tp7sc"
Oct 30 20:26:17.175433 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.175068    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/05f22644-0884-434d-b56b-193d9554b505-image-registry-private-configuration\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:17.175433 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.175117    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ce6c7b90-6b06-4c81-bf95-539b71e2aaed-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf\" (UID: \"ce6c7b90-6b06-4c81-bf95-539b71e2aaed\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf"
Oct 30 20:26:17.175433 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.175122    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv49q\" (UniqueName: \"kubernetes.io/projected/08b2f787-ad25-47f9-8ef8-724308458f88-kube-api-access-hv49q\") pod \"kas-connection-checker-685895c7c9-brf75\" (UID: \"08b2f787-ad25-47f9-8ef8-724308458f88\") " pod="kube-system/kas-connection-checker-685895c7c9-brf75"
Oct 30 20:26:17.175433 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.175163    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzql8\" (UniqueName: \"kubernetes.io/projected/fc3bbdf6-9dad-4519-ae08-faf3cad565e2-kube-api-access-vzql8\") pod \"network-check-source-6f78d59f99-bhrtb\" (UID: \"fc3bbdf6-9dad-4519-ae08-faf3cad565e2\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bhrtb"
Oct 30 20:26:17.175433 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.175188    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d5312b70-82df-4e75-a300-b8e1d63a75f3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9rs22\" (UID: \"d5312b70-82df-4e75-a300-b8e1d63a75f3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9rs22"
Oct 30 20:26:17.175433 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.175230    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/533fb89e-0b69-4c79-9cbf-7baa5f725219-service-ca-bundle\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:17.175433 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.175263    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-bound-sa-token\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:17.175433 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.175275    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9f908129-4500-4433-871c-032c29a55dd9-ca-trust-extracted\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:17.175433 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.175294    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh4r9\" (UniqueName: \"kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-kube-api-access-qh4r9\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:17.175970 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:17.175359    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/533fb89e-0b69-4c79-9cbf-7baa5f725219-service-ca-bundle podName:533fb89e-0b69-4c79-9cbf-7baa5f725219 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:17.675340683 +0000 UTC m=+32.246485757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/533fb89e-0b69-4c79-9cbf-7baa5f725219-service-ca-bundle") pod "router-default-666dd86ff-dnnq5" (UID: "533fb89e-0b69-4c79-9cbf-7baa5f725219") : configmap references non-existent config key: service-ca.crt
Oct 30 20:26:17.175970 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.175403    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94tlh\" (UniqueName: \"kubernetes.io/projected/91fe9e06-0c30-4000-bddb-2ba89cd048cc-kube-api-access-94tlh\") pod \"console-operator-7cd56b8dbc-l8qvb\" (UID: \"91fe9e06-0c30-4000-bddb-2ba89cd048cc\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb"
Oct 30 20:26:17.175970 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.175423    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9f908129-4500-4433-871c-032c29a55dd9-registry-certificates\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:17.175970 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.175439    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf-snapshots\") pod \"insights-operator-7c4986d449-mql5m\" (UID: \"8840f0ca-269e-4924-8b3d-5fe6ff4e90cf\") " pod="openshift-insights/insights-operator-7c4986d449-mql5m"
Oct 30 20:26:17.175970 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.175852    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9f908129-4500-4433-871c-032c29a55dd9-registry-certificates\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:17.178427 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.178399    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ce6c7b90-6b06-4c81-bf95-539b71e2aaed-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf\" (UID: \"ce6c7b90-6b06-4c81-bf95-539b71e2aaed\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf"
Oct 30 20:26:17.179357 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.179266    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 20:26:17.179994 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.179970    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 20:26:17.185030 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.185006    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/91fe9e06-0c30-4000-bddb-2ba89cd048cc-trusted-ca\") pod \"console-operator-7cd56b8dbc-l8qvb\" (UID: \"91fe9e06-0c30-4000-bddb-2ba89cd048cc\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb"
Oct 30 20:26:17.185120 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.185065    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 20:26:17.185174 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.185136    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f908129-4500-4433-871c-032c29a55dd9-trusted-ca\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:17.187627 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.187613    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-5tplx\""
Oct 30 20:26:17.187694 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.187661    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 20:26:17.187694 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.187667    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-d75rn\""
Oct 30 20:26:17.187694 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.187684    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 20:26:17.187816 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.187695    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mkv2p\""
Oct 30 20:26:17.187816 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.187707    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 20:26:17.187816 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.187696    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-dm7dq\""
Oct 30 20:26:17.187816 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.187696    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-vhzdq\""
Oct 30 20:26:17.187816 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.187751    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 20:26:17.188300 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.188283    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 20:26:17.188352 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.188291    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 20:26:17.188352 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.188319    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 20:26:17.188352 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.188329    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 20:26:17.188447 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.188363    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-6lsj6\""
Oct 30 20:26:17.188951 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.188939    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 20:26:17.189016 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.188977    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 20:26:17.189016 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.189003    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nrwxn\""
Oct 30 20:26:17.189120 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.189066    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 20:26:17.189120 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.189093    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 20:26:17.189120 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.189115    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-zhmkz\""
Oct 30 20:26:17.189239 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.189098    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 20:26:17.189239 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.189129    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 20:26:17.189239 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.189154    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-42gkn\""
Oct 30 20:26:17.189239 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.189179    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-lbdrb\""
Oct 30 20:26:17.194630 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:17.194613    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 20:26:17.194700 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:17.194673    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3f7114f-f896-4fc9-b98a-e47574e7720d-samples-operator-tls podName:f3f7114f-f896-4fc9-b98a-e47574e7720d nodeName:}" failed. No retries permitted until 2026-10-30 20:26:17.694656003 +0000 UTC m=+32.265801061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f3f7114f-f896-4fc9-b98a-e47574e7720d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-m9n59" (UID: "f3f7114f-f896-4fc9-b98a-e47574e7720d") : secret "samples-operator-tls" not found
Oct 30 20:26:17.195020 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:17.195001    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:26:17.195020 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:17.195021    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-968d68cdf-vwtgc: secret "image-registry-tls" not found
Oct 30 20:26:17.195304 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:17.195068    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-registry-tls podName:9f908129-4500-4433-871c-032c29a55dd9 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:17.695053238 +0000 UTC m=+32.266198296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-registry-tls") pod "image-registry-968d68cdf-vwtgc" (UID: "9f908129-4500-4433-871c-032c29a55dd9") : secret "image-registry-tls" not found
Oct 30 20:26:17.195747 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:17.195726    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 20:26:17.195842 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:17.195781    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-metrics-certs podName:533fb89e-0b69-4c79-9cbf-7baa5f725219 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:17.695766859 +0000 UTC m=+32.266911924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-metrics-certs") pod "router-default-666dd86ff-dnnq5" (UID: "533fb89e-0b69-4c79-9cbf-7baa5f725219") : secret "router-metrics-certs-default" not found
Oct 30 20:26:17.195842 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:17.195808    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 20:26:17.195950 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:17.195870    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5312b70-82df-4e75-a300-b8e1d63a75f3-cluster-monitoring-operator-tls podName:d5312b70-82df-4e75-a300-b8e1d63a75f3 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:17.695859068 +0000 UTC m=+32.267004128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d5312b70-82df-4e75-a300-b8e1d63a75f3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9rs22" (UID: "d5312b70-82df-4e75-a300-b8e1d63a75f3") : secret "cluster-monitoring-operator-tls" not found
Oct 30 20:26:17.197145 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.197122    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9f908129-4500-4433-871c-032c29a55dd9-image-registry-private-configuration\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:17.197145 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.197137    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-stats-auth\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:17.197306 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.197256    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-default-certificate\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:17.197306 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.197289    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/91fe9e06-0c30-4000-bddb-2ba89cd048cc-serving-cert\") pod \"console-operator-7cd56b8dbc-l8qvb\" (UID: \"91fe9e06-0c30-4000-bddb-2ba89cd048cc\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb"
Oct 30 20:26:17.197447 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.197428    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9f908129-4500-4433-871c-032c29a55dd9-installation-pull-secrets\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:17.199223 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.199203    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-2t45q\""
Oct 30 20:26:17.217640 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.217622    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 20:26:17.217728 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.217674    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 20:26:17.219352 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.219338    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lwgrd\""
Oct 30 20:26:17.220118 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.220100    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 20:26:17.220187 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.220107    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 20:26:17.220187 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.220146    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 20:26:17.220645 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.220630    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-bhrtb"]
Oct 30 20:26:17.221277 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.221262    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z9tdd"]
Oct 30 20:26:17.221894 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.221868    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7db8fb6d4-8n6g7"]
Oct 30 20:26:17.222448 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.222436    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 20:26:17.222571 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.222560    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 20:26:17.222736 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.222724    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bxjhb\""
Oct 30 20:26:17.267890 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.267860    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krgxb\" (UniqueName: \"kubernetes.io/projected/533fb89e-0b69-4c79-9cbf-7baa5f725219-kube-api-access-krgxb\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:17.268028 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.267955    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv49q\" (UniqueName: \"kubernetes.io/projected/08b2f787-ad25-47f9-8ef8-724308458f88-kube-api-access-hv49q\") pod \"kas-connection-checker-685895c7c9-brf75\" (UID: \"08b2f787-ad25-47f9-8ef8-724308458f88\") " pod="kube-system/kas-connection-checker-685895c7c9-brf75"
Oct 30 20:26:17.268174 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.268153    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-bound-sa-token\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:17.268594 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.268568    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsx89\" (UniqueName: \"kubernetes.io/projected/f3f7114f-f896-4fc9-b98a-e47574e7720d-kube-api-access-wsx89\") pod \"cluster-samples-operator-6ddb6c8b94-m9n59\" (UID: \"f3f7114f-f896-4fc9-b98a-e47574e7720d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-m9n59"
Oct 30 20:26:17.268671 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.268591    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tpmm\" (UniqueName: \"kubernetes.io/projected/ce6c7b90-6b06-4c81-bf95-539b71e2aaed-kube-api-access-8tpmm\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf\" (UID: \"ce6c7b90-6b06-4c81-bf95-539b71e2aaed\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf"
Oct 30 20:26:17.269036 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.269010    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8hdc\" (UniqueName: \"kubernetes.io/projected/f4913de8-6db5-413a-938f-abc50a5460e7-kube-api-access-d8hdc\") pod \"kas-connection-checker-685895c7c9-kgsjn\" (UID: \"f4913de8-6db5-413a-938f-abc50a5460e7\") " pod="kube-system/kas-connection-checker-685895c7c9-kgsjn"
Oct 30 20:26:17.269153 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.269095    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wb98\" (UniqueName: \"kubernetes.io/projected/d5312b70-82df-4e75-a300-b8e1d63a75f3-kube-api-access-5wb98\") pod \"cluster-monitoring-operator-884576c57-9rs22\" (UID: \"d5312b70-82df-4e75-a300-b8e1d63a75f3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9rs22"
Oct 30 20:26:17.269341 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.269324    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94tlh\" (UniqueName: \"kubernetes.io/projected/91fe9e06-0c30-4000-bddb-2ba89cd048cc-kube-api-access-94tlh\") pod \"console-operator-7cd56b8dbc-l8qvb\" (UID: \"91fe9e06-0c30-4000-bddb-2ba89cd048cc\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb"
Oct 30 20:26:17.269381 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.269364    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8nqb\" (UniqueName: \"kubernetes.io/projected/78c7e317-9d58-4752-9804-92ac45b5afb7-kube-api-access-m8nqb\") pod \"kas-connection-checker-685895c7c9-c5b2k\" (UID: \"78c7e317-9d58-4752-9804-92ac45b5afb7\") " pod="kube-system/kas-connection-checker-685895c7c9-c5b2k"
Oct 30 20:26:17.270281 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.270263    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqftg\" (UniqueName: \"kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-kube-api-access-nqftg\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:17.275788 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.275768    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-registry-tls\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:17.275884 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.275791    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fxk9\" (UniqueName: \"kubernetes.io/projected/c1e8d4f3-e4a1-416c-bbc7-2c76063bfc24-kube-api-access-7fxk9\") pod \"ingress-canary-mrnvn\" (UID: \"c1e8d4f3-e4a1-416c-bbc7-2c76063bfc24\") " pod="openshift-ingress-canary/ingress-canary-mrnvn"
Oct 30 20:26:17.275884 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.275814    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5040db9b-91d3-4d89-adaa-d1394cc3a617-config\") pod \"service-ca-operator-544d749c8c-tp7sc\" (UID: \"5040db9b-91d3-4d89-adaa-d1394cc3a617\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tp7sc"
Oct 30 20:26:17.275884 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.275845    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/05f22644-0884-434d-b56b-193d9554b505-ca-trust-extracted\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:17.275884 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.275866    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/05f22644-0884-434d-b56b-193d9554b505-trusted-ca\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:17.276111 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.275894    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dx86\" (UniqueName: \"kubernetes.io/projected/a0790a28-d605-4c0c-a652-41ac6ef22e09-kube-api-access-4dx86\") pod \"volume-data-source-validator-58c9f5f46d-z9tdd\" (UID: \"a0790a28-d605-4c0c-a652-41ac6ef22e09\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z9tdd"
Oct 30 20:26:17.276111 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.275929    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5040db9b-91d3-4d89-adaa-d1394cc3a617-serving-cert\") pod \"service-ca-operator-544d749c8c-tp7sc\" (UID: \"5040db9b-91d3-4d89-adaa-d1394cc3a617\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tp7sc"
Oct 30 20:26:17.276111 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:17.275933    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:26:17.276111 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:17.275951    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7db8fb6d4-8n6g7: secret "image-registry-tls" not found
Oct 30 20:26:17.276111 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.275953    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/05f22644-0884-434d-b56b-193d9554b505-image-registry-private-configuration\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:17.276111 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.275988    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzql8\" (UniqueName: \"kubernetes.io/projected/fc3bbdf6-9dad-4519-ae08-faf3cad565e2-kube-api-access-vzql8\") pod \"network-check-source-6f78d59f99-bhrtb\" (UID: \"fc3bbdf6-9dad-4519-ae08-faf3cad565e2\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bhrtb"
Oct 30 20:26:17.276111 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:17.276016    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-registry-tls podName:05f22644-0884-434d-b56b-193d9554b505 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:17.775998854 +0000 UTC m=+32.347143927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-registry-tls") pod "image-registry-7db8fb6d4-8n6g7" (UID: "05f22644-0884-434d-b56b-193d9554b505") : secret "image-registry-tls" not found
Oct 30 20:26:17.276111 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.276080    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh4r9\" (UniqueName: \"kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-kube-api-access-qh4r9\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:17.276111 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.276118    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf-snapshots\") pod \"insights-operator-7c4986d449-mql5m\" (UID: \"8840f0ca-269e-4924-8b3d-5fe6ff4e90cf\") " pod="openshift-insights/insights-operator-7c4986d449-mql5m"
Oct 30 20:26:17.276594 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.276164    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgf7l\" (UniqueName: \"kubernetes.io/projected/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-kube-api-access-zgf7l\") pod \"dns-default-h2vh7\" (UID: \"4c7e75b7-c3cd-4791-9d39-9120f6fd95dc\") " pod="openshift-dns/dns-default-h2vh7"
Oct 30 20:26:17.276594 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.276188    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-bound-sa-token\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:17.276594 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.276193    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/05f22644-0884-434d-b56b-193d9554b505-ca-trust-extracted\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:17.276594 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.276217    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/05f22644-0884-434d-b56b-193d9554b505-registry-certificates\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:17.276594 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.276369    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5040db9b-91d3-4d89-adaa-d1394cc3a617-config\") pod \"service-ca-operator-544d749c8c-tp7sc\" (UID: \"5040db9b-91d3-4d89-adaa-d1394cc3a617\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tp7sc"
Oct 30 20:26:17.276884 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.276637    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/05f22644-0884-434d-b56b-193d9554b505-installation-pull-secrets\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:17.276884 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.276686    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-tmp-dir\") pod \"dns-default-h2vh7\" (UID: \"4c7e75b7-c3cd-4791-9d39-9120f6fd95dc\") " pod="openshift-dns/dns-default-h2vh7"
Oct 30 20:26:17.276884 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.276714    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf-serving-cert\") pod \"insights-operator-7c4986d449-mql5m\" (UID: \"8840f0ca-269e-4924-8b3d-5fe6ff4e90cf\") " pod="openshift-insights/insights-operator-7c4986d449-mql5m"
Oct 30 20:26:17.276884 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.276728    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/05f22644-0884-434d-b56b-193d9554b505-registry-certificates\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:17.276884 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.276745    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/70fa9600-c497-4deb-a133-3cb24f1c8050-nginx-conf\") pod \"networking-console-plugin-86c478bb88-4v8mv\" (UID: \"70fa9600-c497-4deb-a133-3cb24f1c8050\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4v8mv"
Oct 30 20:26:17.277110 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.277075    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-tmp-dir\") pod \"dns-default-h2vh7\" (UID: \"4c7e75b7-c3cd-4791-9d39-9120f6fd95dc\") " pod="openshift-dns/dns-default-h2vh7"
Oct 30 20:26:17.277162 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.277116    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsmc8\" (UniqueName: \"kubernetes.io/projected/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf-kube-api-access-tsmc8\") pod \"insights-operator-7c4986d449-mql5m\" (UID: \"8840f0ca-269e-4924-8b3d-5fe6ff4e90cf\") " pod="openshift-insights/insights-operator-7c4986d449-mql5m"
Oct 30 20:26:17.277214 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.277164    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-metrics-tls\") pod \"dns-default-h2vh7\" (UID: \"4c7e75b7-c3cd-4791-9d39-9120f6fd95dc\") " pod="openshift-dns/dns-default-h2vh7"
Oct 30 20:26:17.277214 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.277208    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-config-volume\") pod \"dns-default-h2vh7\" (UID: \"4c7e75b7-c3cd-4791-9d39-9120f6fd95dc\") " pod="openshift-dns/dns-default-h2vh7"
Oct 30 20:26:17.277407 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.277244    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70fa9600-c497-4deb-a133-3cb24f1c8050-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4v8mv\" (UID: \"70fa9600-c497-4deb-a133-3cb24f1c8050\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4v8mv"
Oct 30 20:26:17.277407 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.277269    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf-tmp\") pod \"insights-operator-7c4986d449-mql5m\" (UID: \"8840f0ca-269e-4924-8b3d-5fe6ff4e90cf\") " pod="openshift-insights/insights-operator-7c4986d449-mql5m"
Oct 30 20:26:17.277407 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.277293    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-mql5m\" (UID: \"8840f0ca-269e-4924-8b3d-5fe6ff4e90cf\") " pod="openshift-insights/insights-operator-7c4986d449-mql5m"
Oct 30 20:26:17.277407 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.277383    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64rvp\" (UniqueName: \"kubernetes.io/projected/5040db9b-91d3-4d89-adaa-d1394cc3a617-kube-api-access-64rvp\") pod \"service-ca-operator-544d749c8c-tp7sc\" (UID: \"5040db9b-91d3-4d89-adaa-d1394cc3a617\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tp7sc"
Oct 30 20:26:17.277670 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.277497    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1e8d4f3-e4a1-416c-bbc7-2c76063bfc24-cert\") pod \"ingress-canary-mrnvn\" (UID: \"c1e8d4f3-e4a1-416c-bbc7-2c76063bfc24\") " pod="openshift-ingress-canary/ingress-canary-mrnvn"
Oct 30 20:26:17.277670 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.277547    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf-service-ca-bundle\") pod \"insights-operator-7c4986d449-mql5m\" (UID: \"8840f0ca-269e-4924-8b3d-5fe6ff4e90cf\") " pod="openshift-insights/insights-operator-7c4986d449-mql5m"
Oct 30 20:26:17.277781 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:17.277721    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:26:17.277781 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:17.277771    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-metrics-tls podName:4c7e75b7-c3cd-4791-9d39-9120f6fd95dc nodeName:}" failed. No retries permitted until 2026-10-30 20:26:17.777755702 +0000 UTC m=+32.348900775 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-metrics-tls") pod "dns-default-h2vh7" (UID: "4c7e75b7-c3cd-4791-9d39-9120f6fd95dc") : secret "dns-default-metrics-tls" not found
Oct 30 20:26:17.279530 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.278023    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf-snapshots\") pod \"insights-operator-7c4986d449-mql5m\" (UID: \"8840f0ca-269e-4924-8b3d-5fe6ff4e90cf\") " pod="openshift-insights/insights-operator-7c4986d449-mql5m"
Oct 30 20:26:17.279530 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.278051    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-mql5m\" (UID: \"8840f0ca-269e-4924-8b3d-5fe6ff4e90cf\") " pod="openshift-insights/insights-operator-7c4986d449-mql5m"
Oct 30 20:26:17.279530 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:17.278444    2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 20:26:17.279530 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.278491    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf-service-ca-bundle\") pod \"insights-operator-7c4986d449-mql5m\" (UID: \"8840f0ca-269e-4924-8b3d-5fe6ff4e90cf\") " pod="openshift-insights/insights-operator-7c4986d449-mql5m"
Oct 30 20:26:17.279530 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:17.278519    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70fa9600-c497-4deb-a133-3cb24f1c8050-networking-console-plugin-cert podName:70fa9600-c497-4deb-a133-3cb24f1c8050 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:17.778480265 +0000 UTC m=+32.349625322 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/70fa9600-c497-4deb-a133-3cb24f1c8050-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-4v8mv" (UID: "70fa9600-c497-4deb-a133-3cb24f1c8050") : secret "networking-console-plugin-cert" not found
Oct 30 20:26:17.279530 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.278613    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf"
Oct 30 20:26:17.279530 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.278768    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-config-volume\") pod \"dns-default-h2vh7\" (UID: \"4c7e75b7-c3cd-4791-9d39-9120f6fd95dc\") " pod="openshift-dns/dns-default-h2vh7"
Oct 30 20:26:17.279530 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.278901    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf-tmp\") pod \"insights-operator-7c4986d449-mql5m\" (UID: \"8840f0ca-269e-4924-8b3d-5fe6ff4e90cf\") " pod="openshift-insights/insights-operator-7c4986d449-mql5m"
Oct 30 20:26:17.279530 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.279416    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/70fa9600-c497-4deb-a133-3cb24f1c8050-nginx-conf\") pod \"networking-console-plugin-86c478bb88-4v8mv\" (UID: \"70fa9600-c497-4deb-a133-3cb24f1c8050\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4v8mv"
Oct 30 20:26:17.280237 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.279948    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/05f22644-0884-434d-b56b-193d9554b505-trusted-ca\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:17.281006 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.280986    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf-serving-cert\") pod \"insights-operator-7c4986d449-mql5m\" (UID: \"8840f0ca-269e-4924-8b3d-5fe6ff4e90cf\") " pod="openshift-insights/insights-operator-7c4986d449-mql5m"
Oct 30 20:26:17.281093 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.280985    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/05f22644-0884-434d-b56b-193d9554b505-image-registry-private-configuration\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:17.281144 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.281114    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5040db9b-91d3-4d89-adaa-d1394cc3a617-serving-cert\") pod \"service-ca-operator-544d749c8c-tp7sc\" (UID: \"5040db9b-91d3-4d89-adaa-d1394cc3a617\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tp7sc"
Oct 30 20:26:17.281144 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.281136    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1e8d4f3-e4a1-416c-bbc7-2c76063bfc24-cert\") pod \"ingress-canary-mrnvn\" (UID: \"c1e8d4f3-e4a1-416c-bbc7-2c76063bfc24\") " pod="openshift-ingress-canary/ingress-canary-mrnvn"
Oct 30 20:26:17.281498 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.281483    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/05f22644-0884-434d-b56b-193d9554b505-installation-pull-secrets\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:17.308526 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.308506    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-brf75"
Oct 30 20:26:17.313597 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.313579    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-c5b2k"
Oct 30 20:26:17.352945 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.352229    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzql8\" (UniqueName: \"kubernetes.io/projected/fc3bbdf6-9dad-4519-ae08-faf3cad565e2-kube-api-access-vzql8\") pod \"network-check-source-6f78d59f99-bhrtb\" (UID: \"fc3bbdf6-9dad-4519-ae08-faf3cad565e2\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bhrtb"
Oct 30 20:26:17.352945 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.352890    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsmc8\" (UniqueName: \"kubernetes.io/projected/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf-kube-api-access-tsmc8\") pod \"insights-operator-7c4986d449-mql5m\" (UID: \"8840f0ca-269e-4924-8b3d-5fe6ff4e90cf\") " pod="openshift-insights/insights-operator-7c4986d449-mql5m"
Oct 30 20:26:17.353646 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.353589    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgf7l\" (UniqueName: \"kubernetes.io/projected/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-kube-api-access-zgf7l\") pod \"dns-default-h2vh7\" (UID: \"4c7e75b7-c3cd-4791-9d39-9120f6fd95dc\") " pod="openshift-dns/dns-default-h2vh7"
Oct 30 20:26:17.354468 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.354076    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64rvp\" (UniqueName: \"kubernetes.io/projected/5040db9b-91d3-4d89-adaa-d1394cc3a617-kube-api-access-64rvp\") pod \"service-ca-operator-544d749c8c-tp7sc\" (UID: \"5040db9b-91d3-4d89-adaa-d1394cc3a617\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tp7sc"
Oct 30 20:26:17.354468 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.354439    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dx86\" (UniqueName: \"kubernetes.io/projected/a0790a28-d605-4c0c-a652-41ac6ef22e09-kube-api-access-4dx86\") pod \"volume-data-source-validator-58c9f5f46d-z9tdd\" (UID: \"a0790a28-d605-4c0c-a652-41ac6ef22e09\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z9tdd"
Oct 30 20:26:17.355221 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.355188    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-bound-sa-token\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:17.355434 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.355414    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh4r9\" (UniqueName: \"kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-kube-api-access-qh4r9\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:17.371852 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.371787    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fxk9\" (UniqueName: \"kubernetes.io/projected/c1e8d4f3-e4a1-416c-bbc7-2c76063bfc24-kube-api-access-7fxk9\") pod \"ingress-canary-mrnvn\" (UID: \"c1e8d4f3-e4a1-416c-bbc7-2c76063bfc24\") " pod="openshift-ingress-canary/ingress-canary-mrnvn"
Oct 30 20:26:17.376859 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.376635    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tp7sc"
Oct 30 20:26:17.402194 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.401942    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z9tdd"
Oct 30 20:26:17.430788 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.430576    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-mql5m"
Oct 30 20:26:17.438180 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.438149    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bhrtb"
Oct 30 20:26:17.442524 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.442330    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb"
Oct 30 20:26:17.450978 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.448153    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mrnvn"
Oct 30 20:26:17.569332 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.569103    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-kgsjn"
Oct 30 20:26:17.682234 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.682207    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/533fb89e-0b69-4c79-9cbf-7baa5f725219-service-ca-bundle\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:17.682368 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:17.682325    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/533fb89e-0b69-4c79-9cbf-7baa5f725219-service-ca-bundle podName:533fb89e-0b69-4c79-9cbf-7baa5f725219 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:18.68231189 +0000 UTC m=+33.253456947 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/533fb89e-0b69-4c79-9cbf-7baa5f725219-service-ca-bundle") pod "router-default-666dd86ff-dnnq5" (UID: "533fb89e-0b69-4c79-9cbf-7baa5f725219") : configmap references non-existent config key: service-ca.crt
Oct 30 20:26:17.728430 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.728378    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf"]
Oct 30 20:26:17.731887 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.731853    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-c5b2k"]
Oct 30 20:26:17.732023 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:26:17.732001    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce6c7b90_6b06_4c81_bf95_539b71e2aaed.slice/crio-74f6fb01bca34a911a5b4074647d79ef03b18d23b55e713aca7d8b452344a7ae WatchSource:0}: Error finding container 74f6fb01bca34a911a5b4074647d79ef03b18d23b55e713aca7d8b452344a7ae: Status 404 returned error can't find the container with id 74f6fb01bca34a911a5b4074647d79ef03b18d23b55e713aca7d8b452344a7ae
Oct 30 20:26:17.732727 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.732629    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-brf75"]
Oct 30 20:26:17.734195 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:26:17.734155    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08b2f787_ad25_47f9_8ef8_724308458f88.slice/crio-8a1271576ec3b8c9517dd5d184a1f8c2c1b2c19190b24ba6ef7ae3a9cf805ea4 WatchSource:0}: Error finding container 8a1271576ec3b8c9517dd5d184a1f8c2c1b2c19190b24ba6ef7ae3a9cf805ea4: Status 404 returned error can't find the container with id 8a1271576ec3b8c9517dd5d184a1f8c2c1b2c19190b24ba6ef7ae3a9cf805ea4
Oct 30 20:26:17.734953 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:26:17.734931    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c7e317_9d58_4752_9804_92ac45b5afb7.slice/crio-f0cc702f1f07050383c2518b83f462e244a58ce3c3c9a431fdb7f186a57dce6d WatchSource:0}: Error finding container f0cc702f1f07050383c2518b83f462e244a58ce3c3c9a431fdb7f186a57dce6d: Status 404 returned error can't find the container with id f0cc702f1f07050383c2518b83f462e244a58ce3c3c9a431fdb7f186a57dce6d
Oct 30 20:26:17.780214 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.779136    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-tp7sc"]
Oct 30 20:26:17.784474 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.783052    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-metrics-tls\") pod \"dns-default-h2vh7\" (UID: \"4c7e75b7-c3cd-4791-9d39-9120f6fd95dc\") " pod="openshift-dns/dns-default-h2vh7"
Oct 30 20:26:17.784474 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.783104    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70fa9600-c497-4deb-a133-3cb24f1c8050-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4v8mv\" (UID: \"70fa9600-c497-4deb-a133-3cb24f1c8050\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4v8mv"
Oct 30 20:26:17.784474 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.783141    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f3f7114f-f896-4fc9-b98a-e47574e7720d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-m9n59\" (UID: \"f3f7114f-f896-4fc9-b98a-e47574e7720d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-m9n59"
Oct 30 20:26:17.784474 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.783184    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-registry-tls\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:17.784474 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.783210    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-registry-tls\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:17.784474 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.783258    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-metrics-certs\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:17.784474 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.783295    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d5312b70-82df-4e75-a300-b8e1d63a75f3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9rs22\" (UID: \"d5312b70-82df-4e75-a300-b8e1d63a75f3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9rs22"
Oct 30 20:26:17.784474 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:17.783468    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 20:26:17.784474 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:17.783531    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5312b70-82df-4e75-a300-b8e1d63a75f3-cluster-monitoring-operator-tls podName:d5312b70-82df-4e75-a300-b8e1d63a75f3 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:18.78351351 +0000 UTC m=+33.354658579 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d5312b70-82df-4e75-a300-b8e1d63a75f3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9rs22" (UID: "d5312b70-82df-4e75-a300-b8e1d63a75f3") : secret "cluster-monitoring-operator-tls" not found
Oct 30 20:26:17.784474 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:17.783596    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:26:17.784474 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:17.783636    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-metrics-tls podName:4c7e75b7-c3cd-4791-9d39-9120f6fd95dc nodeName:}" failed. No retries permitted until 2026-10-30 20:26:18.783616534 +0000 UTC m=+33.354761608 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-metrics-tls") pod "dns-default-h2vh7" (UID: "4c7e75b7-c3cd-4791-9d39-9120f6fd95dc") : secret "dns-default-metrics-tls" not found
Oct 30 20:26:17.784474 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:17.783687    2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 20:26:17.784474 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:17.783717    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70fa9600-c497-4deb-a133-3cb24f1c8050-networking-console-plugin-cert podName:70fa9600-c497-4deb-a133-3cb24f1c8050 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:18.783707165 +0000 UTC m=+33.354852223 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/70fa9600-c497-4deb-a133-3cb24f1c8050-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-4v8mv" (UID: "70fa9600-c497-4deb-a133-3cb24f1c8050") : secret "networking-console-plugin-cert" not found
Oct 30 20:26:17.784474 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:17.783766    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 20:26:17.784474 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:17.783795    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3f7114f-f896-4fc9-b98a-e47574e7720d-samples-operator-tls podName:f3f7114f-f896-4fc9-b98a-e47574e7720d nodeName:}" failed. No retries permitted until 2026-10-30 20:26:18.783786695 +0000 UTC m=+33.354931766 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f3f7114f-f896-4fc9-b98a-e47574e7720d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-m9n59" (UID: "f3f7114f-f896-4fc9-b98a-e47574e7720d") : secret "samples-operator-tls" not found
Oct 30 20:26:17.784474 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:17.783876    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:26:17.785413 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:17.783897    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-968d68cdf-vwtgc: secret "image-registry-tls" not found
Oct 30 20:26:17.785413 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:17.783928    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-registry-tls podName:9f908129-4500-4433-871c-032c29a55dd9 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:18.78391805 +0000 UTC m=+33.355063121 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-registry-tls") pod "image-registry-968d68cdf-vwtgc" (UID: "9f908129-4500-4433-871c-032c29a55dd9") : secret "image-registry-tls" not found
Oct 30 20:26:17.785413 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:17.783982    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:26:17.785413 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:17.783991    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7db8fb6d4-8n6g7: secret "image-registry-tls" not found
Oct 30 20:26:17.785413 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:17.784018    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-registry-tls podName:05f22644-0884-434d-b56b-193d9554b505 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:18.784008742 +0000 UTC m=+33.355153812 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-registry-tls") pod "image-registry-7db8fb6d4-8n6g7" (UID: "05f22644-0884-434d-b56b-193d9554b505") : secret "image-registry-tls" not found
Oct 30 20:26:17.785413 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:17.784068    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 20:26:17.785413 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:17.784101    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-metrics-certs podName:533fb89e-0b69-4c79-9cbf-7baa5f725219 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:18.784092345 +0000 UTC m=+33.355237418 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-metrics-certs") pod "router-default-666dd86ff-dnnq5" (UID: "533fb89e-0b69-4c79-9cbf-7baa5f725219") : secret "router-metrics-certs-default" not found
Oct 30 20:26:17.843570 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.843175    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-mql5m"]
Oct 30 20:26:17.846790 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:26:17.846108    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8840f0ca_269e_4924_8b3d_5fe6ff4e90cf.slice/crio-30316f990be28a3c33df9b2c59fd5df78eb099b0430afed2ca8b4801cd011ff7 WatchSource:0}: Error finding container 30316f990be28a3c33df9b2c59fd5df78eb099b0430afed2ca8b4801cd011ff7: Status 404 returned error can't find the container with id 30316f990be28a3c33df9b2c59fd5df78eb099b0430afed2ca8b4801cd011ff7
Oct 30 20:26:17.846790 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.846627    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mrnvn"]
Oct 30 20:26:17.850457 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:26:17.850437    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1e8d4f3_e4a1_416c_bbc7_2c76063bfc24.slice/crio-bbba1dc4118eac14545671fe4575ef65fb40fdd43ef94fba6642f7afcea5810e WatchSource:0}: Error finding container bbba1dc4118eac14545671fe4575ef65fb40fdd43ef94fba6642f7afcea5810e: Status 404 returned error can't find the container with id bbba1dc4118eac14545671fe4575ef65fb40fdd43ef94fba6642f7afcea5810e
Oct 30 20:26:17.854573 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.854492    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-l8qvb"]
Oct 30 20:26:17.856923 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:26:17.856900    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91fe9e06_0c30_4000_bddb_2ba89cd048cc.slice/crio-50c63a4bd7251e656a4eaea02805ff737213b40bee52f7fe6c2b4e73e32e21a8 WatchSource:0}: Error finding container 50c63a4bd7251e656a4eaea02805ff737213b40bee52f7fe6c2b4e73e32e21a8: Status 404 returned error can't find the container with id 50c63a4bd7251e656a4eaea02805ff737213b40bee52f7fe6c2b4e73e32e21a8
Oct 30 20:26:17.875697 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.875658    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-bhrtb"]
Oct 30 20:26:17.877480 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:26:17.877461    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc3bbdf6_9dad_4519_ae08_faf3cad565e2.slice/crio-a56836a111808a0d4adaf9c71b33bc6390b76a15aedd4c96b5c54ec31f364f88 WatchSource:0}: Error finding container a56836a111808a0d4adaf9c71b33bc6390b76a15aedd4c96b5c54ec31f364f88: Status 404 returned error can't find the container with id a56836a111808a0d4adaf9c71b33bc6390b76a15aedd4c96b5c54ec31f364f88
Oct 30 20:26:17.920262 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.920238    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z9tdd"]
Oct 30 20:26:17.923311 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:26:17.923290    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0790a28_d605_4c0c_a652_41ac6ef22e09.slice/crio-38fc45f90c3b66e3c7d6b46a31d34de20e8ba9732b4a3b5841ea69328045fbc5 WatchSource:0}: Error finding container 38fc45f90c3b66e3c7d6b46a31d34de20e8ba9732b4a3b5841ea69328045fbc5: Status 404 returned error can't find the container with id 38fc45f90c3b66e3c7d6b46a31d34de20e8ba9732b4a3b5841ea69328045fbc5
Oct 30 20:26:17.960282 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:17.960255    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kgsjn"]
Oct 30 20:26:17.962080 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:26:17.962058    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4913de8_6db5_413a_938f_abc50a5460e7.slice/crio-fdd0cdaef89fda618b32966daf0807210809105824091b7c26732c5089116bf5 WatchSource:0}: Error finding container fdd0cdaef89fda618b32966daf0807210809105824091b7c26732c5089116bf5: Status 404 returned error can't find the container with id fdd0cdaef89fda618b32966daf0807210809105824091b7c26732c5089116bf5
Oct 30 20:26:18.170724 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:18.170640    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb" event={"ID":"91fe9e06-0c30-4000-bddb-2ba89cd048cc","Type":"ContainerStarted","Data":"50c63a4bd7251e656a4eaea02805ff737213b40bee52f7fe6c2b4e73e32e21a8"}
Oct 30 20:26:18.171807 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:18.171781    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bhrtb" event={"ID":"fc3bbdf6-9dad-4519-ae08-faf3cad565e2","Type":"ContainerStarted","Data":"a56836a111808a0d4adaf9c71b33bc6390b76a15aedd4c96b5c54ec31f364f88"}
Oct 30 20:26:18.172883 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:18.172862    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z9tdd" event={"ID":"a0790a28-d605-4c0c-a652-41ac6ef22e09","Type":"ContainerStarted","Data":"38fc45f90c3b66e3c7d6b46a31d34de20e8ba9732b4a3b5841ea69328045fbc5"}
Oct 30 20:26:18.174226 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:18.174203    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-c5b2k" event={"ID":"78c7e317-9d58-4752-9804-92ac45b5afb7","Type":"ContainerStarted","Data":"3000e8f98b8a323257ecd94eac5c20e8bf9d9855b10864eb267b6e9a77ad4217"}
Oct 30 20:26:18.174293 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:18.174237    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-c5b2k" event={"ID":"78c7e317-9d58-4752-9804-92ac45b5afb7","Type":"ContainerStarted","Data":"f0cc702f1f07050383c2518b83f462e244a58ce3c3c9a431fdb7f186a57dce6d"}
Oct 30 20:26:18.175443 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:18.175415    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mrnvn" event={"ID":"c1e8d4f3-e4a1-416c-bbc7-2c76063bfc24","Type":"ContainerStarted","Data":"bbba1dc4118eac14545671fe4575ef65fb40fdd43ef94fba6642f7afcea5810e"}
Oct 30 20:26:18.176504 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:18.176485    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-mql5m" event={"ID":"8840f0ca-269e-4924-8b3d-5fe6ff4e90cf","Type":"ContainerStarted","Data":"30316f990be28a3c33df9b2c59fd5df78eb099b0430afed2ca8b4801cd011ff7"}
Oct 30 20:26:18.177858 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:18.177834    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-kgsjn" event={"ID":"f4913de8-6db5-413a-938f-abc50a5460e7","Type":"ContainerStarted","Data":"18fa028b8eecf57f7f83686380738894b41cfd7314faa7f5620d77c96ceff971"}
Oct 30 20:26:18.177942 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:18.177878    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-kgsjn" event={"ID":"f4913de8-6db5-413a-938f-abc50a5460e7","Type":"ContainerStarted","Data":"fdd0cdaef89fda618b32966daf0807210809105824091b7c26732c5089116bf5"}
Oct 30 20:26:18.179004 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:18.178983    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tp7sc" event={"ID":"5040db9b-91d3-4d89-adaa-d1394cc3a617","Type":"ContainerStarted","Data":"24bfadd2814fc8daad30a6323ebc6fe65849fec0b2d2c1689868fc5fa2e0ada1"}
Oct 30 20:26:18.180382 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:18.180356    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-brf75" event={"ID":"08b2f787-ad25-47f9-8ef8-724308458f88","Type":"ContainerStarted","Data":"9e605bc84321105fe252fec5628bc317912164b3de0f3d38ffd3e7d9d37ddf15"}
Oct 30 20:26:18.180458 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:18.180390    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-brf75" event={"ID":"08b2f787-ad25-47f9-8ef8-724308458f88","Type":"ContainerStarted","Data":"8a1271576ec3b8c9517dd5d184a1f8c2c1b2c19190b24ba6ef7ae3a9cf805ea4"}
Oct 30 20:26:18.181399 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:18.181378    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf" event={"ID":"ce6c7b90-6b06-4c81-bf95-539b71e2aaed","Type":"ContainerStarted","Data":"74f6fb01bca34a911a5b4074647d79ef03b18d23b55e713aca7d8b452344a7ae"}
Oct 30 20:26:18.691922 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:18.691889    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/533fb89e-0b69-4c79-9cbf-7baa5f725219-service-ca-bundle\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:18.692089 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:18.692074    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/533fb89e-0b69-4c79-9cbf-7baa5f725219-service-ca-bundle podName:533fb89e-0b69-4c79-9cbf-7baa5f725219 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:20.692058498 +0000 UTC m=+35.263203571 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/533fb89e-0b69-4c79-9cbf-7baa5f725219-service-ca-bundle") pod "router-default-666dd86ff-dnnq5" (UID: "533fb89e-0b69-4c79-9cbf-7baa5f725219") : configmap references non-existent config key: service-ca.crt
Oct 30 20:26:18.792408 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:18.792364    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-metrics-tls\") pod \"dns-default-h2vh7\" (UID: \"4c7e75b7-c3cd-4791-9d39-9120f6fd95dc\") " pod="openshift-dns/dns-default-h2vh7"
Oct 30 20:26:18.792408 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:18.792408    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70fa9600-c497-4deb-a133-3cb24f1c8050-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4v8mv\" (UID: \"70fa9600-c497-4deb-a133-3cb24f1c8050\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4v8mv"
Oct 30 20:26:18.792599 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:18.792440    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f3f7114f-f896-4fc9-b98a-e47574e7720d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-m9n59\" (UID: \"f3f7114f-f896-4fc9-b98a-e47574e7720d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-m9n59"
Oct 30 20:26:18.792599 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:18.792477    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-registry-tls\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:18.792599 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:18.792517    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-registry-tls\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:18.792599 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:18.792533    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:26:18.792599 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:18.792545    2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 20:26:18.792599 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:18.792596    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:26:18.792871 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:18.792608    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7db8fb6d4-8n6g7: secret "image-registry-tls" not found
Oct 30 20:26:18.792871 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:18.792608    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 20:26:18.792871 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:18.792622    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:26:18.792871 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:18.792637    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-968d68cdf-vwtgc: secret "image-registry-tls" not found
Oct 30 20:26:18.792871 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:18.792613    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-metrics-tls podName:4c7e75b7-c3cd-4791-9d39-9120f6fd95dc nodeName:}" failed. No retries permitted until 2026-10-30 20:26:20.792595084 +0000 UTC m=+35.363740157 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-metrics-tls") pod "dns-default-h2vh7" (UID: "4c7e75b7-c3cd-4791-9d39-9120f6fd95dc") : secret "dns-default-metrics-tls" not found
Oct 30 20:26:18.792871 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:18.792722    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-metrics-certs\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:18.792871 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:18.792761    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70fa9600-c497-4deb-a133-3cb24f1c8050-networking-console-plugin-cert podName:70fa9600-c497-4deb-a133-3cb24f1c8050 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:20.792748468 +0000 UTC m=+35.363893524 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/70fa9600-c497-4deb-a133-3cb24f1c8050-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-4v8mv" (UID: "70fa9600-c497-4deb-a133-3cb24f1c8050") : secret "networking-console-plugin-cert" not found
Oct 30 20:26:18.792871 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:18.792781    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-registry-tls podName:05f22644-0884-434d-b56b-193d9554b505 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:20.792775732 +0000 UTC m=+35.363920789 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-registry-tls") pod "image-registry-7db8fb6d4-8n6g7" (UID: "05f22644-0884-434d-b56b-193d9554b505") : secret "image-registry-tls" not found
Oct 30 20:26:18.792871 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:18.792789    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 20:26:18.792871 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:18.792799    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3f7114f-f896-4fc9-b98a-e47574e7720d-samples-operator-tls podName:f3f7114f-f896-4fc9-b98a-e47574e7720d nodeName:}" failed. No retries permitted until 2026-10-30 20:26:20.792794334 +0000 UTC m=+35.363939391 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f3f7114f-f896-4fc9-b98a-e47574e7720d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-m9n59" (UID: "f3f7114f-f896-4fc9-b98a-e47574e7720d") : secret "samples-operator-tls" not found
Oct 30 20:26:18.792871 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:18.792809    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-registry-tls podName:9f908129-4500-4433-871c-032c29a55dd9 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:20.792804652 +0000 UTC m=+35.363949708 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-registry-tls") pod "image-registry-968d68cdf-vwtgc" (UID: "9f908129-4500-4433-871c-032c29a55dd9") : secret "image-registry-tls" not found
Oct 30 20:26:18.792871 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:18.792850    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d5312b70-82df-4e75-a300-b8e1d63a75f3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9rs22\" (UID: \"d5312b70-82df-4e75-a300-b8e1d63a75f3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9rs22"
Oct 30 20:26:18.793263 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:18.792880    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-metrics-certs podName:533fb89e-0b69-4c79-9cbf-7baa5f725219 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:20.792861752 +0000 UTC m=+35.364006823 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-metrics-certs") pod "router-default-666dd86ff-dnnq5" (UID: "533fb89e-0b69-4c79-9cbf-7baa5f725219") : secret "router-metrics-certs-default" not found
Oct 30 20:26:18.793263 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:18.792918    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 20:26:18.793263 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:18.792961    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5312b70-82df-4e75-a300-b8e1d63a75f3-cluster-monitoring-operator-tls podName:d5312b70-82df-4e75-a300-b8e1d63a75f3 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:20.792949108 +0000 UTC m=+35.364094164 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d5312b70-82df-4e75-a300-b8e1d63a75f3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9rs22" (UID: "d5312b70-82df-4e75-a300-b8e1d63a75f3") : secret "cluster-monitoring-operator-tls" not found
Oct 30 20:26:18.893778 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:18.893727    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/88d9611e-c7ed-490b-b401-530d1649c873-original-pull-secret\") pod \"global-pull-secret-syncer-bqh9l\" (UID: \"88d9611e-c7ed-490b-b401-530d1649c873\") " pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:26:18.894243 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:18.893789    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/214f4f91-4d6e-46e0-ac1f-d23935cd7215-metrics-certs\") pod \"network-metrics-daemon-zcslg\" (UID: \"214f4f91-4d6e-46e0-ac1f-d23935cd7215\") " pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:26:18.894243 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:18.894023    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 20:26:18.894243 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:18.894085    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/214f4f91-4d6e-46e0-ac1f-d23935cd7215-metrics-certs podName:214f4f91-4d6e-46e0-ac1f-d23935cd7215 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:50.894065908 +0000 UTC m=+65.465210984 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/214f4f91-4d6e-46e0-ac1f-d23935cd7215-metrics-certs") pod "network-metrics-daemon-zcslg" (UID: "214f4f91-4d6e-46e0-ac1f-d23935cd7215") : secret "metrics-daemon-secret" not found
Oct 30 20:26:18.905596 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:18.905571    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/88d9611e-c7ed-490b-b401-530d1649c873-original-pull-secret\") pod \"global-pull-secret-syncer-bqh9l\" (UID: \"88d9611e-c7ed-490b-b401-530d1649c873\") " pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:26:18.940508 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:18.940450    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-kgsjn" podStartSLOduration=389.940431979 podStartE2EDuration="6m29.940431979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:18.939028656 +0000 UTC m=+33.510173735" watchObservedRunningTime="2026-10-30 20:26:18.940431979 +0000 UTC m=+33.511577060"
Oct 30 20:26:18.994441 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:18.994377    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb792\" (UniqueName: \"kubernetes.io/projected/c9732ee5-c72e-44c6-9917-49cc9f81ffe0-kube-api-access-bb792\") pod \"network-check-target-pfvnn\" (UID: \"c9732ee5-c72e-44c6-9917-49cc9f81ffe0\") " pod="openshift-network-diagnostics/network-check-target-pfvnn"
Oct 30 20:26:18.999644 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:18.999616    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb792\" (UniqueName: \"kubernetes.io/projected/c9732ee5-c72e-44c6-9917-49cc9f81ffe0-kube-api-access-bb792\") pod \"network-check-target-pfvnn\" (UID: \"c9732ee5-c72e-44c6-9917-49cc9f81ffe0\") " pod="openshift-network-diagnostics/network-check-target-pfvnn"
Oct 30 20:26:19.096025 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:19.095733    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfvnn"
Oct 30 20:26:19.102279 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:19.102247    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:26:19.168851 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:19.168179    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-c5b2k" podStartSLOduration=390.168165301 podStartE2EDuration="6m30.168165301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:19.167899211 +0000 UTC m=+33.739044287" watchObservedRunningTime="2026-10-30 20:26:19.168165301 +0000 UTC m=+33.739310383"
Oct 30 20:26:19.168851 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:19.168311    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-brf75" podStartSLOduration=390.168304009 podStartE2EDuration="6m30.168304009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:19.054916689 +0000 UTC m=+33.626061769" watchObservedRunningTime="2026-10-30 20:26:19.168304009 +0000 UTC m=+33.739449088"
Oct 30 20:26:19.499858 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:19.499445    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:19.530477 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:19.530447    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pfvnn"]
Oct 30 20:26:19.546912 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:19.546888    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bqh9l"]
Oct 30 20:26:19.571878 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:26:19.551874    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d9611e_c7ed_490b_b401_530d1649c873.slice/crio-a655a658b97ac67d1a4601390a3b06ce0901e4bd12c56182b337145e476ad060 WatchSource:0}: Error finding container a655a658b97ac67d1a4601390a3b06ce0901e4bd12c56182b337145e476ad060: Status 404 returned error can't find the container with id a655a658b97ac67d1a4601390a3b06ce0901e4bd12c56182b337145e476ad060
Oct 30 20:26:19.729241 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:19.729205    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:20.135116 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:20.134281    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:20.198425 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:20.198351    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bqh9l" event={"ID":"88d9611e-c7ed-490b-b401-530d1649c873","Type":"ContainerStarted","Data":"a655a658b97ac67d1a4601390a3b06ce0901e4bd12c56182b337145e476ad060"}
Oct 30 20:26:20.204155 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:20.204123    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pfvnn" event={"ID":"c9732ee5-c72e-44c6-9917-49cc9f81ffe0","Type":"ContainerStarted","Data":"9f982210565e5738a437f8b5a6bea6e6a28dfdbfe4fadaa5e4828dde56b8f30c"}
Oct 30 20:26:20.713170 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:20.713134    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/533fb89e-0b69-4c79-9cbf-7baa5f725219-service-ca-bundle\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:20.713500 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:20.713482    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/533fb89e-0b69-4c79-9cbf-7baa5f725219-service-ca-bundle podName:533fb89e-0b69-4c79-9cbf-7baa5f725219 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:24.713460465 +0000 UTC m=+39.284605535 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/533fb89e-0b69-4c79-9cbf-7baa5f725219-service-ca-bundle") pod "router-default-666dd86ff-dnnq5" (UID: "533fb89e-0b69-4c79-9cbf-7baa5f725219") : configmap references non-existent config key: service-ca.crt
Oct 30 20:26:20.813787 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:20.813747    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d5312b70-82df-4e75-a300-b8e1d63a75f3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9rs22\" (UID: \"d5312b70-82df-4e75-a300-b8e1d63a75f3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9rs22"
Oct 30 20:26:20.813974 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:20.813878    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-metrics-tls\") pod \"dns-default-h2vh7\" (UID: \"4c7e75b7-c3cd-4791-9d39-9120f6fd95dc\") " pod="openshift-dns/dns-default-h2vh7"
Oct 30 20:26:20.813974 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:20.813919    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70fa9600-c497-4deb-a133-3cb24f1c8050-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4v8mv\" (UID: \"70fa9600-c497-4deb-a133-3cb24f1c8050\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4v8mv"
Oct 30 20:26:20.813974 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:20.813951    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f3f7114f-f896-4fc9-b98a-e47574e7720d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-m9n59\" (UID: \"f3f7114f-f896-4fc9-b98a-e47574e7720d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-m9n59"
Oct 30 20:26:20.814138 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:20.813981    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-registry-tls\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:20.814138 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:20.813998    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-registry-tls\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:20.814138 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:20.814037    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-metrics-certs\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:20.814281 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:20.814168    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 20:26:20.814281 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:20.814226    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-metrics-certs podName:533fb89e-0b69-4c79-9cbf-7baa5f725219 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:24.814213191 +0000 UTC m=+39.385358248 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-metrics-certs") pod "router-default-666dd86ff-dnnq5" (UID: "533fb89e-0b69-4c79-9cbf-7baa5f725219") : secret "router-metrics-certs-default" not found
Oct 30 20:26:20.814658 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:20.814604    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 20:26:20.814765 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:20.814663    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5312b70-82df-4e75-a300-b8e1d63a75f3-cluster-monitoring-operator-tls podName:d5312b70-82df-4e75-a300-b8e1d63a75f3 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:24.814647515 +0000 UTC m=+39.385792578 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d5312b70-82df-4e75-a300-b8e1d63a75f3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9rs22" (UID: "d5312b70-82df-4e75-a300-b8e1d63a75f3") : secret "cluster-monitoring-operator-tls" not found
Oct 30 20:26:20.814765 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:20.814722    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:26:20.814765 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:20.814749    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-metrics-tls podName:4c7e75b7-c3cd-4791-9d39-9120f6fd95dc nodeName:}" failed. No retries permitted until 2026-10-30 20:26:24.814740651 +0000 UTC m=+39.385885708 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-metrics-tls") pod "dns-default-h2vh7" (UID: "4c7e75b7-c3cd-4791-9d39-9120f6fd95dc") : secret "dns-default-metrics-tls" not found
Oct 30 20:26:20.814891 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:20.814780    2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 20:26:20.814891 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:20.814810    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70fa9600-c497-4deb-a133-3cb24f1c8050-networking-console-plugin-cert podName:70fa9600-c497-4deb-a133-3cb24f1c8050 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:24.814800451 +0000 UTC m=+39.385945511 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/70fa9600-c497-4deb-a133-3cb24f1c8050-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-4v8mv" (UID: "70fa9600-c497-4deb-a133-3cb24f1c8050") : secret "networking-console-plugin-cert" not found
Oct 30 20:26:20.814891 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:20.814864    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 20:26:20.814979 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:20.814885    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3f7114f-f896-4fc9-b98a-e47574e7720d-samples-operator-tls podName:f3f7114f-f896-4fc9-b98a-e47574e7720d nodeName:}" failed. No retries permitted until 2026-10-30 20:26:24.814878999 +0000 UTC m=+39.386024057 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f3f7114f-f896-4fc9-b98a-e47574e7720d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-m9n59" (UID: "f3f7114f-f896-4fc9-b98a-e47574e7720d") : secret "samples-operator-tls" not found
Oct 30 20:26:20.814979 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:20.814933    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:26:20.814979 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:20.814940    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-968d68cdf-vwtgc: secret "image-registry-tls" not found
Oct 30 20:26:20.814979 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:20.814959    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-registry-tls podName:9f908129-4500-4433-871c-032c29a55dd9 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:24.814953381 +0000 UTC m=+39.386098437 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-registry-tls") pod "image-registry-968d68cdf-vwtgc" (UID: "9f908129-4500-4433-871c-032c29a55dd9") : secret "image-registry-tls" not found
Oct 30 20:26:20.815207 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:20.814999    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:26:20.815207 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:20.815005    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7db8fb6d4-8n6g7: secret "image-registry-tls" not found
Oct 30 20:26:20.815207 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:20.815021    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-registry-tls podName:05f22644-0884-434d-b56b-193d9554b505 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:24.815016038 +0000 UTC m=+39.386161095 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-registry-tls") pod "image-registry-7db8fb6d4-8n6g7" (UID: "05f22644-0884-434d-b56b-193d9554b505") : secret "image-registry-tls" not found
Oct 30 20:26:21.212240 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:21.212164    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z9tdd" event={"ID":"a0790a28-d605-4c0c-a652-41ac6ef22e09","Type":"ContainerStarted","Data":"79f3713360a0a5a1b17cfaf903f39162043a7388e59355899a4292fc4f33911d"}
Oct 30 20:26:21.328410 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:21.328342    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z9tdd" podStartSLOduration=327.420796306 podStartE2EDuration="5m30.328322979s" totalImagesPullingTime="2.907526673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:21.326980735 +0000 UTC m=+35.898125806" watchObservedRunningTime="2026-10-30 20:26:21.328322979 +0000 UTC m=+35.899468059"
Oct 30 20:26:21.469776 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:21.469692    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:21.629408 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:21.629373    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:24.031356 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:24.031321    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:24.191390 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:24.191352    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:24.225966 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:24.225892    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-mql5m" event={"ID":"8840f0ca-269e-4924-8b3d-5fe6ff4e90cf","Type":"ContainerStarted","Data":"6d6e8f3dc531658573169c482a26a7fcf917aec85fd5fc380ccd5a3d7c266197"}
Oct 30 20:26:24.229593 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:24.229564    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tp7sc" event={"ID":"5040db9b-91d3-4d89-adaa-d1394cc3a617","Type":"ContainerStarted","Data":"b7ecbaf4df570e688022c9d51b06a367bdbc0160489cf748651b668cb5d7e585"}
Oct 30 20:26:24.232538 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:24.232511    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf" event={"ID":"ce6c7b90-6b06-4c81-bf95-539b71e2aaed","Type":"ContainerStarted","Data":"b8586be1520b81fb122e2d0c3f13e8c1fa8fefef1764be88b7ff904c090e57f6"}
Oct 30 20:26:24.234885 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:24.234862    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8qvb_91fe9e06-0c30-4000-bddb-2ba89cd048cc/console-operator/0.log"
Oct 30 20:26:24.235008 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:24.234897    2575 generic.go:356] "Generic (PLEG): container finished" podID="91fe9e06-0c30-4000-bddb-2ba89cd048cc" containerID="a1d7a6ec8bf71cf468cbd0e0b2d4a150cddb4d1d9174ce5658f7c3e4e576e393" exitCode=255
Oct 30 20:26:24.235008 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:24.234958    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb" event={"ID":"91fe9e06-0c30-4000-bddb-2ba89cd048cc","Type":"ContainerDied","Data":"a1d7a6ec8bf71cf468cbd0e0b2d4a150cddb4d1d9174ce5658f7c3e4e576e393"}
Oct 30 20:26:24.235217 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:24.235202    2575 scope.go:122] "RemoveContainer" containerID="a1d7a6ec8bf71cf468cbd0e0b2d4a150cddb4d1d9174ce5658f7c3e4e576e393"
Oct 30 20:26:24.468860 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:24.468778    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf" podStartSLOduration=335.82692158 podStartE2EDuration="5m41.468759687s" totalImagesPullingTime="5.641838107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:24.468186147 +0000 UTC m=+39.039331225" watchObservedRunningTime="2026-10-30 20:26:24.468759687 +0000 UTC m=+39.039904769"
Oct 30 20:26:24.469730 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:24.469695    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-mql5m" podStartSLOduration=326.368744046 podStartE2EDuration="5m32.469681957s" totalImagesPullingTime="6.100937911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:24.347033673 +0000 UTC m=+38.918178754" watchObservedRunningTime="2026-10-30 20:26:24.469681957 +0000 UTC m=+39.040827039"
Oct 30 20:26:24.584055 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:24.583992    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tp7sc" podStartSLOduration=330.407701812 podStartE2EDuration="5m36.583970771s" totalImagesPullingTime="6.176268959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:24.582430412 +0000 UTC m=+39.153575503" watchObservedRunningTime="2026-10-30 20:26:24.583970771 +0000 UTC m=+39.155115850"
Oct 30 20:26:24.761072 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:24.761029    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/533fb89e-0b69-4c79-9cbf-7baa5f725219-service-ca-bundle\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:24.761408 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:24.761391    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/533fb89e-0b69-4c79-9cbf-7baa5f725219-service-ca-bundle podName:533fb89e-0b69-4c79-9cbf-7baa5f725219 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:32.761374899 +0000 UTC m=+47.332519977 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/533fb89e-0b69-4c79-9cbf-7baa5f725219-service-ca-bundle") pod "router-default-666dd86ff-dnnq5" (UID: "533fb89e-0b69-4c79-9cbf-7baa5f725219") : configmap references non-existent config key: service-ca.crt
Oct 30 20:26:24.871238 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:24.871199    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-metrics-tls\") pod \"dns-default-h2vh7\" (UID: \"4c7e75b7-c3cd-4791-9d39-9120f6fd95dc\") " pod="openshift-dns/dns-default-h2vh7"
Oct 30 20:26:24.871354 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:24.871254    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70fa9600-c497-4deb-a133-3cb24f1c8050-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4v8mv\" (UID: \"70fa9600-c497-4deb-a133-3cb24f1c8050\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4v8mv"
Oct 30 20:26:24.871354 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:24.871290    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f3f7114f-f896-4fc9-b98a-e47574e7720d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-m9n59\" (UID: \"f3f7114f-f896-4fc9-b98a-e47574e7720d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-m9n59"
Oct 30 20:26:24.871354 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:24.871324    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-registry-tls\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:24.871354 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:24.871347    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-registry-tls\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:24.871559 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:24.871379    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-metrics-certs\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:24.871559 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:24.871413    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d5312b70-82df-4e75-a300-b8e1d63a75f3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9rs22\" (UID: \"d5312b70-82df-4e75-a300-b8e1d63a75f3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9rs22"
Oct 30 20:26:24.871559 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:24.871428    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:26:24.871559 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:24.871499    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-metrics-tls podName:4c7e75b7-c3cd-4791-9d39-9120f6fd95dc nodeName:}" failed. No retries permitted until 2026-10-30 20:26:32.871478573 +0000 UTC m=+47.442623634 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-metrics-tls") pod "dns-default-h2vh7" (UID: "4c7e75b7-c3cd-4791-9d39-9120f6fd95dc") : secret "dns-default-metrics-tls" not found
Oct 30 20:26:24.871559 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:24.871555    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 20:26:24.873633 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:24.871577    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:26:24.873633 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:24.871590    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-968d68cdf-vwtgc: secret "image-registry-tls" not found
Oct 30 20:26:24.873633 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:24.871602    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5312b70-82df-4e75-a300-b8e1d63a75f3-cluster-monitoring-operator-tls podName:d5312b70-82df-4e75-a300-b8e1d63a75f3 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:32.871588357 +0000 UTC m=+47.442733421 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d5312b70-82df-4e75-a300-b8e1d63a75f3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9rs22" (UID: "d5312b70-82df-4e75-a300-b8e1d63a75f3") : secret "cluster-monitoring-operator-tls" not found
Oct 30 20:26:24.873633 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:24.871628    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-registry-tls podName:9f908129-4500-4433-871c-032c29a55dd9 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:32.871616192 +0000 UTC m=+47.442761252 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-registry-tls") pod "image-registry-968d68cdf-vwtgc" (UID: "9f908129-4500-4433-871c-032c29a55dd9") : secret "image-registry-tls" not found
Oct 30 20:26:24.873633 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:24.871665    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:26:24.873633 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:24.871675    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7db8fb6d4-8n6g7: secret "image-registry-tls" not found
Oct 30 20:26:24.873633 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:24.871684    2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 20:26:24.873633 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:24.871703    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-registry-tls podName:05f22644-0884-434d-b56b-193d9554b505 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:32.871693441 +0000 UTC m=+47.442838502 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-registry-tls") pod "image-registry-7db8fb6d4-8n6g7" (UID: "05f22644-0884-434d-b56b-193d9554b505") : secret "image-registry-tls" not found
Oct 30 20:26:24.873633 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:24.871717    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70fa9600-c497-4deb-a133-3cb24f1c8050-networking-console-plugin-cert podName:70fa9600-c497-4deb-a133-3cb24f1c8050 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:32.871708934 +0000 UTC m=+47.442853994 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/70fa9600-c497-4deb-a133-3cb24f1c8050-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-4v8mv" (UID: "70fa9600-c497-4deb-a133-3cb24f1c8050") : secret "networking-console-plugin-cert" not found
Oct 30 20:26:24.873633 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:24.871753    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 20:26:24.873633 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:24.871783    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3f7114f-f896-4fc9-b98a-e47574e7720d-samples-operator-tls podName:f3f7114f-f896-4fc9-b98a-e47574e7720d nodeName:}" failed. No retries permitted until 2026-10-30 20:26:32.871772376 +0000 UTC m=+47.442917437 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f3f7114f-f896-4fc9-b98a-e47574e7720d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-m9n59" (UID: "f3f7114f-f896-4fc9-b98a-e47574e7720d") : secret "samples-operator-tls" not found
Oct 30 20:26:24.873633 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:24.871938    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 20:26:24.873633 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:24.871978    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-metrics-certs podName:533fb89e-0b69-4c79-9cbf-7baa5f725219 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:32.871967446 +0000 UTC m=+47.443112543 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-metrics-certs") pod "router-default-666dd86ff-dnnq5" (UID: "533fb89e-0b69-4c79-9cbf-7baa5f725219") : secret "router-metrics-certs-default" not found
Oct 30 20:26:25.108019 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.107957    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:25.242264 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.242234    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8qvb_91fe9e06-0c30-4000-bddb-2ba89cd048cc/console-operator/1.log"
Oct 30 20:26:25.242799 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.242783    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8qvb_91fe9e06-0c30-4000-bddb-2ba89cd048cc/console-operator/0.log"
Oct 30 20:26:25.242914 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.242852    2575 generic.go:356] "Generic (PLEG): container finished" podID="91fe9e06-0c30-4000-bddb-2ba89cd048cc" containerID="b21ea8949c9de8f71605d238975ce27033db9c0ec4dff0ed0d0894a5dbcff357" exitCode=255
Oct 30 20:26:25.242972 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.242943    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb" event={"ID":"91fe9e06-0c30-4000-bddb-2ba89cd048cc","Type":"ContainerDied","Data":"b21ea8949c9de8f71605d238975ce27033db9c0ec4dff0ed0d0894a5dbcff357"}
Oct 30 20:26:25.243152 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.243044    2575 scope.go:122] "RemoveContainer" containerID="a1d7a6ec8bf71cf468cbd0e0b2d4a150cddb4d1d9174ce5658f7c3e4e576e393"
Oct 30 20:26:25.243312 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.243286    2575 scope.go:122] "RemoveContainer" containerID="b21ea8949c9de8f71605d238975ce27033db9c0ec4dff0ed0d0894a5dbcff357"
Oct 30 20:26:25.243513 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:25.243490    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-l8qvb_openshift-console-operator(91fe9e06-0c30-4000-bddb-2ba89cd048cc)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb" podUID="91fe9e06-0c30-4000-bddb-2ba89cd048cc"
Oct 30 20:26:25.244724 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.244679    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mrnvn" event={"ID":"c1e8d4f3-e4a1-416c-bbc7-2c76063bfc24","Type":"ContainerStarted","Data":"0f0850a68de25969424e3689fa596b75658fceca3b2bdaa7fbba37a21e367b56"}
Oct 30 20:26:25.249982 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.249959    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lfrbc"]
Oct 30 20:26:25.265550 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.264936    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lfrbc"
Oct 30 20:26:25.265550 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.265411    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-hbm7v"]
Oct 30 20:26:25.291747 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.291276    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hbm7v"
Oct 30 20:26:25.292338 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.291870    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:25.308369 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.307111    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tk6nb"]
Oct 30 20:26:25.319700 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.319672    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-qcj9t\""
Oct 30 20:26:25.323018 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.322912    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 20:26:25.323153 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.323131    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 20:26:25.334510 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.334487    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lfrbc"]
Oct 30 20:26:25.334637 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.334532    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-hbm7v"]
Oct 30 20:26:25.334637 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.334613    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:25.376253 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.375846    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tk6nb"]
Oct 30 20:26:25.376253 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.375933    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6kxw\" (UniqueName: \"kubernetes.io/projected/fe578b91-829c-450a-a60a-0f4e4b6e8c63-kube-api-access-m6kxw\") pod \"migrator-66df5d496b-lfrbc\" (UID: \"fe578b91-829c-450a-a60a-0f4e4b6e8c63\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lfrbc"
Oct 30 20:26:25.420254 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.420200    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bpqv7\""
Oct 30 20:26:25.420254 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.420205    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 20:26:25.422737 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.422686    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 20:26:25.459544 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.459465    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:25.477439 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.477411    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b87eabe7-2403-4534-987b-ce6ecc6fda94-data-volume\") pod \"insights-runtime-extractor-tk6nb\" (UID: \"b87eabe7-2403-4534-987b-ce6ecc6fda94\") " pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:25.478717 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.477463    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b87eabe7-2403-4534-987b-ce6ecc6fda94-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tk6nb\" (UID: \"b87eabe7-2403-4534-987b-ce6ecc6fda94\") " pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:25.478717 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.477604    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b87eabe7-2403-4534-987b-ce6ecc6fda94-crio-socket\") pod \"insights-runtime-extractor-tk6nb\" (UID: \"b87eabe7-2403-4534-987b-ce6ecc6fda94\") " pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:25.478717 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.477645    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss27z\" (UniqueName: \"kubernetes.io/projected/50f563bf-3336-49c6-9708-041c41b41c53-kube-api-access-ss27z\") pod \"migrator-66df5d496b-hbm7v\" (UID: \"50f563bf-3336-49c6-9708-041c41b41c53\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hbm7v"
Oct 30 20:26:25.478717 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.477745    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2gcf\" (UniqueName: \"kubernetes.io/projected/b87eabe7-2403-4534-987b-ce6ecc6fda94-kube-api-access-g2gcf\") pod \"insights-runtime-extractor-tk6nb\" (UID: \"b87eabe7-2403-4534-987b-ce6ecc6fda94\") " pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:25.478717 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.477779    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6kxw\" (UniqueName: \"kubernetes.io/projected/fe578b91-829c-450a-a60a-0f4e4b6e8c63-kube-api-access-m6kxw\") pod \"migrator-66df5d496b-lfrbc\" (UID: \"fe578b91-829c-450a-a60a-0f4e4b6e8c63\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lfrbc"
Oct 30 20:26:25.478717 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.477961    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b87eabe7-2403-4534-987b-ce6ecc6fda94-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tk6nb\" (UID: \"b87eabe7-2403-4534-987b-ce6ecc6fda94\") " pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:25.538164 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.538128    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6kxw\" (UniqueName: \"kubernetes.io/projected/fe578b91-829c-450a-a60a-0f4e4b6e8c63-kube-api-access-m6kxw\") pod \"migrator-66df5d496b-lfrbc\" (UID: \"fe578b91-829c-450a-a60a-0f4e4b6e8c63\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lfrbc"
Oct 30 20:26:25.545353 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.545326    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-zmknv-22np8"]
Oct 30 20:26:25.569086 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.568683    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-zmknv-22np8"
Oct 30 20:26:25.578561 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.578530    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b87eabe7-2403-4534-987b-ce6ecc6fda94-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tk6nb\" (UID: \"b87eabe7-2403-4534-987b-ce6ecc6fda94\") " pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:25.578696 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.578623    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b87eabe7-2403-4534-987b-ce6ecc6fda94-crio-socket\") pod \"insights-runtime-extractor-tk6nb\" (UID: \"b87eabe7-2403-4534-987b-ce6ecc6fda94\") " pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:25.578696 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.578653    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss27z\" (UniqueName: \"kubernetes.io/projected/50f563bf-3336-49c6-9708-041c41b41c53-kube-api-access-ss27z\") pod \"migrator-66df5d496b-hbm7v\" (UID: \"50f563bf-3336-49c6-9708-041c41b41c53\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hbm7v"
Oct 30 20:26:25.578696 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:25.578665    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 20:26:25.578696 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.578682    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2gcf\" (UniqueName: \"kubernetes.io/projected/b87eabe7-2403-4534-987b-ce6ecc6fda94-kube-api-access-g2gcf\") pod \"insights-runtime-extractor-tk6nb\" (UID: \"b87eabe7-2403-4534-987b-ce6ecc6fda94\") " pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:25.578931 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:25.578731    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b87eabe7-2403-4534-987b-ce6ecc6fda94-kube-rbac-proxy-cm podName:b87eabe7-2403-4534-987b-ce6ecc6fda94 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:26.078712371 +0000 UTC m=+40.649857451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b87eabe7-2403-4534-987b-ce6ecc6fda94-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tk6nb" (UID: "b87eabe7-2403-4534-987b-ce6ecc6fda94") : configmap "kube-rbac-proxy" not found
Oct 30 20:26:25.578931 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.578842    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b87eabe7-2403-4534-987b-ce6ecc6fda94-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tk6nb\" (UID: \"b87eabe7-2403-4534-987b-ce6ecc6fda94\") " pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:25.579035 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.578943    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b87eabe7-2403-4534-987b-ce6ecc6fda94-crio-socket\") pod \"insights-runtime-extractor-tk6nb\" (UID: \"b87eabe7-2403-4534-987b-ce6ecc6fda94\") " pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:25.579035 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.578950    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b87eabe7-2403-4534-987b-ce6ecc6fda94-data-volume\") pod \"insights-runtime-extractor-tk6nb\" (UID: \"b87eabe7-2403-4534-987b-ce6ecc6fda94\") " pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:25.579151 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:25.579064    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:26:25.579151 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:25.579108    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b87eabe7-2403-4534-987b-ce6ecc6fda94-insights-runtime-extractor-tls podName:b87eabe7-2403-4534-987b-ce6ecc6fda94 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:26.079094771 +0000 UTC m=+40.650239864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b87eabe7-2403-4534-987b-ce6ecc6fda94-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tk6nb" (UID: "b87eabe7-2403-4534-987b-ce6ecc6fda94") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:26:25.583102 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.583079    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b87eabe7-2403-4534-987b-ce6ecc6fda94-data-volume\") pod \"insights-runtime-extractor-tk6nb\" (UID: \"b87eabe7-2403-4534-987b-ce6ecc6fda94\") " pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:25.600671 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.600638    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lfrbc"
Oct 30 20:26:25.610299 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.610249    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mrnvn" podStartSLOduration=2.677045594 podStartE2EDuration="9.610234983s" totalImagesPullingTime="6.933189389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:25.608573851 +0000 UTC m=+40.179718931" watchObservedRunningTime="2026-10-30 20:26:25.610234983 +0000 UTC m=+40.181380095"
Oct 30 20:26:25.612764 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.612704    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-zmknv-22np8"]
Oct 30 20:26:25.646839 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.639979    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2gcf\" (UniqueName: \"kubernetes.io/projected/b87eabe7-2403-4534-987b-ce6ecc6fda94-kube-api-access-g2gcf\") pod \"insights-runtime-extractor-tk6nb\" (UID: \"b87eabe7-2403-4534-987b-ce6ecc6fda94\") " pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:25.646839 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.644183    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:25.647037 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.647013    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss27z\" (UniqueName: \"kubernetes.io/projected/50f563bf-3336-49c6-9708-041c41b41c53-kube-api-access-ss27z\") pod \"migrator-66df5d496b-hbm7v\" (UID: \"50f563bf-3336-49c6-9708-041c41b41c53\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hbm7v"
Oct 30 20:26:25.681425 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.681308    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee380d52-b71c-4839-8835-078d98a5e0ee-service-ca-bundle\") pod \"periodic-gathering-zmknv-22np8\" (UID: \"ee380d52-b71c-4839-8835-078d98a5e0ee\") " pod="openshift-insights/periodic-gathering-zmknv-22np8"
Oct 30 20:26:25.682024 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.681685    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l2sr\" (UniqueName: \"kubernetes.io/projected/ee380d52-b71c-4839-8835-078d98a5e0ee-kube-api-access-9l2sr\") pod \"periodic-gathering-zmknv-22np8\" (UID: \"ee380d52-b71c-4839-8835-078d98a5e0ee\") " pod="openshift-insights/periodic-gathering-zmknv-22np8"
Oct 30 20:26:25.682024 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.681739    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ee380d52-b71c-4839-8835-078d98a5e0ee-archives-path\") pod \"periodic-gathering-zmknv-22np8\" (UID: \"ee380d52-b71c-4839-8835-078d98a5e0ee\") " pod="openshift-insights/periodic-gathering-zmknv-22np8"
Oct 30 20:26:25.682024 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.681814    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee380d52-b71c-4839-8835-078d98a5e0ee-trusted-ca-bundle\") pod \"periodic-gathering-zmknv-22np8\" (UID: \"ee380d52-b71c-4839-8835-078d98a5e0ee\") " pod="openshift-insights/periodic-gathering-zmknv-22np8"
Oct 30 20:26:25.782763 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.782723    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee380d52-b71c-4839-8835-078d98a5e0ee-service-ca-bundle\") pod \"periodic-gathering-zmknv-22np8\" (UID: \"ee380d52-b71c-4839-8835-078d98a5e0ee\") " pod="openshift-insights/periodic-gathering-zmknv-22np8"
Oct 30 20:26:25.782999 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.782891    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l2sr\" (UniqueName: \"kubernetes.io/projected/ee380d52-b71c-4839-8835-078d98a5e0ee-kube-api-access-9l2sr\") pod \"periodic-gathering-zmknv-22np8\" (UID: \"ee380d52-b71c-4839-8835-078d98a5e0ee\") " pod="openshift-insights/periodic-gathering-zmknv-22np8"
Oct 30 20:26:25.782999 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.782984    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ee380d52-b71c-4839-8835-078d98a5e0ee-archives-path\") pod \"periodic-gathering-zmknv-22np8\" (UID: \"ee380d52-b71c-4839-8835-078d98a5e0ee\") " pod="openshift-insights/periodic-gathering-zmknv-22np8"
Oct 30 20:26:25.783112 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.783061    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee380d52-b71c-4839-8835-078d98a5e0ee-trusted-ca-bundle\") pod \"periodic-gathering-zmknv-22np8\" (UID: \"ee380d52-b71c-4839-8835-078d98a5e0ee\") " pod="openshift-insights/periodic-gathering-zmknv-22np8"
Oct 30 20:26:25.783686 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.783660    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ee380d52-b71c-4839-8835-078d98a5e0ee-archives-path\") pod \"periodic-gathering-zmknv-22np8\" (UID: \"ee380d52-b71c-4839-8835-078d98a5e0ee\") " pod="openshift-insights/periodic-gathering-zmknv-22np8"
Oct 30 20:26:25.784101 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.784053    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee380d52-b71c-4839-8835-078d98a5e0ee-service-ca-bundle\") pod \"periodic-gathering-zmknv-22np8\" (UID: \"ee380d52-b71c-4839-8835-078d98a5e0ee\") " pod="openshift-insights/periodic-gathering-zmknv-22np8"
Oct 30 20:26:25.784454 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.784421    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee380d52-b71c-4839-8835-078d98a5e0ee-trusted-ca-bundle\") pod \"periodic-gathering-zmknv-22np8\" (UID: \"ee380d52-b71c-4839-8835-078d98a5e0ee\") " pod="openshift-insights/periodic-gathering-zmknv-22np8"
Oct 30 20:26:25.849035 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.849004    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l2sr\" (UniqueName: \"kubernetes.io/projected/ee380d52-b71c-4839-8835-078d98a5e0ee-kube-api-access-9l2sr\") pod \"periodic-gathering-zmknv-22np8\" (UID: \"ee380d52-b71c-4839-8835-078d98a5e0ee\") " pod="openshift-insights/periodic-gathering-zmknv-22np8"
Oct 30 20:26:25.867956 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.867925    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:25.905177 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.904920    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-zmknv-22np8"
Oct 30 20:26:25.922707 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:25.922676    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hbm7v"
Oct 30 20:26:26.031235 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:26.031155    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lfrbc"]
Oct 30 20:26:26.035064 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:26:26.035039    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe578b91_829c_450a_a60a_0f4e4b6e8c63.slice/crio-8887ae7a21f88fa23debe43d42535d7b88646ed33ceb9eab7ef9c33c38edc394 WatchSource:0}: Error finding container 8887ae7a21f88fa23debe43d42535d7b88646ed33ceb9eab7ef9c33c38edc394: Status 404 returned error can't find the container with id 8887ae7a21f88fa23debe43d42535d7b88646ed33ceb9eab7ef9c33c38edc394
Oct 30 20:26:26.083509 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:26.083378    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:26.089686 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:26.089652    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b87eabe7-2403-4534-987b-ce6ecc6fda94-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tk6nb\" (UID: \"b87eabe7-2403-4534-987b-ce6ecc6fda94\") " pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:26.089871 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:26.089787    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:26:26.089871 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:26.089865    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b87eabe7-2403-4534-987b-ce6ecc6fda94-insights-runtime-extractor-tls podName:b87eabe7-2403-4534-987b-ce6ecc6fda94 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:27.089849898 +0000 UTC m=+41.660994955 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b87eabe7-2403-4534-987b-ce6ecc6fda94-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tk6nb" (UID: "b87eabe7-2403-4534-987b-ce6ecc6fda94") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:26:26.089988 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:26.089873    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 20:26:26.089988 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:26.089940    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b87eabe7-2403-4534-987b-ce6ecc6fda94-kube-rbac-proxy-cm podName:b87eabe7-2403-4534-987b-ce6ecc6fda94 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:27.0899143 +0000 UTC m=+41.661059370 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b87eabe7-2403-4534-987b-ce6ecc6fda94-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tk6nb" (UID: "b87eabe7-2403-4534-987b-ce6ecc6fda94") : configmap "kube-rbac-proxy" not found
Oct 30 20:26:26.089988 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:26.089782    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b87eabe7-2403-4534-987b-ce6ecc6fda94-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tk6nb\" (UID: \"b87eabe7-2403-4534-987b-ce6ecc6fda94\") " pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:26.250408 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:26.250335    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:26.250992 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:26.250966    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bhrtb" event={"ID":"fc3bbdf6-9dad-4519-ae08-faf3cad565e2","Type":"ContainerStarted","Data":"11e1ba871b34ea600e527356ee47f1585f012acf3cd921b639a8c911ef31f315"}
Oct 30 20:26:26.252777 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:26.252752    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8qvb_91fe9e06-0c30-4000-bddb-2ba89cd048cc/console-operator/1.log"
Oct 30 20:26:26.254058 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:26.253162    2575 scope.go:122] "RemoveContainer" containerID="b21ea8949c9de8f71605d238975ce27033db9c0ec4dff0ed0d0894a5dbcff357"
Oct 30 20:26:26.254058 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:26.253389    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-l8qvb_openshift-console-operator(91fe9e06-0c30-4000-bddb-2ba89cd048cc)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb" podUID="91fe9e06-0c30-4000-bddb-2ba89cd048cc"
Oct 30 20:26:26.254310 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:26.254291    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lfrbc" event={"ID":"fe578b91-829c-450a-a60a-0f4e4b6e8c63","Type":"ContainerStarted","Data":"8887ae7a21f88fa23debe43d42535d7b88646ed33ceb9eab7ef9c33c38edc394"}
Oct 30 20:26:26.316657 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:26.316622    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-zmknv-22np8"]
Oct 30 20:26:26.321488 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:26:26.321462    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee380d52_b71c_4839_8835_078d98a5e0ee.slice/crio-3ce759cd9d7ce2099096ca884b37156c6e61e84eae795065d82f7c8b5d1a3d18 WatchSource:0}: Error finding container 3ce759cd9d7ce2099096ca884b37156c6e61e84eae795065d82f7c8b5d1a3d18: Status 404 returned error can't find the container with id 3ce759cd9d7ce2099096ca884b37156c6e61e84eae795065d82f7c8b5d1a3d18
Oct 30 20:26:26.336706 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:26.336674    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-hbm7v"]
Oct 30 20:26:26.339267 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:26:26.339243    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f563bf_3336_49c6_9708_041c41b41c53.slice/crio-9fe27797e37f4642def3994fea218e7ddb7c94d7f571fff9b06c0f5066ae7aba WatchSource:0}: Error finding container 9fe27797e37f4642def3994fea218e7ddb7c94d7f571fff9b06c0f5066ae7aba: Status 404 returned error can't find the container with id 9fe27797e37f4642def3994fea218e7ddb7c94d7f571fff9b06c0f5066ae7aba
Oct 30 20:26:26.368216 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:26.368164    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bhrtb" podStartSLOduration=336.08399739 podStartE2EDuration="5m43.36814498s" totalImagesPullingTime="7.28414759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:26.367928797 +0000 UTC m=+40.939073878" watchObservedRunningTime="2026-10-30 20:26:26.36814498 +0000 UTC m=+40.939290062"
Oct 30 20:26:27.099476 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:27.099442    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b87eabe7-2403-4534-987b-ce6ecc6fda94-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tk6nb\" (UID: \"b87eabe7-2403-4534-987b-ce6ecc6fda94\") " pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:27.099664 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:27.099591    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b87eabe7-2403-4534-987b-ce6ecc6fda94-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tk6nb\" (UID: \"b87eabe7-2403-4534-987b-ce6ecc6fda94\") " pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:27.099726 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:27.099712    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:26:27.099779 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:27.099772    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b87eabe7-2403-4534-987b-ce6ecc6fda94-insights-runtime-extractor-tls podName:b87eabe7-2403-4534-987b-ce6ecc6fda94 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:29.099754138 +0000 UTC m=+43.670899212 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b87eabe7-2403-4534-987b-ce6ecc6fda94-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tk6nb" (UID: "b87eabe7-2403-4534-987b-ce6ecc6fda94") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:26:27.100164 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:27.100137    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 20:26:27.100260 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:27.100196    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b87eabe7-2403-4534-987b-ce6ecc6fda94-kube-rbac-proxy-cm podName:b87eabe7-2403-4534-987b-ce6ecc6fda94 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:29.100179924 +0000 UTC m=+43.671324995 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b87eabe7-2403-4534-987b-ce6ecc6fda94-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tk6nb" (UID: "b87eabe7-2403-4534-987b-ce6ecc6fda94") : configmap "kube-rbac-proxy" not found
Oct 30 20:26:27.134990 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:27.134956    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-6b75b"]
Oct 30 20:26:27.137591 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:27.137521    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-6b75b"
Oct 30 20:26:27.192570 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:27.192538    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 20:26:27.192733 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:27.192693    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-mgc68\""
Oct 30 20:26:27.193328 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:27.193307    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 20:26:27.193431 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:27.193331    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 20:26:27.195198 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:27.195178    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 20:26:27.197257 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:27.197235    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-6b75b"]
Oct 30 20:26:27.200997 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:27.200972    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/309f81fb-9b40-4053-a060-8672a7c84f77-signing-key\") pod \"service-ca-5fffc97bdf-6b75b\" (UID: \"309f81fb-9b40-4053-a060-8672a7c84f77\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6b75b"
Oct 30 20:26:27.201126 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:27.201104    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/309f81fb-9b40-4053-a060-8672a7c84f77-signing-cabundle\") pod \"service-ca-5fffc97bdf-6b75b\" (UID: \"309f81fb-9b40-4053-a060-8672a7c84f77\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6b75b"
Oct 30 20:26:27.201248 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:27.201164    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnvlx\" (UniqueName: \"kubernetes.io/projected/309f81fb-9b40-4053-a060-8672a7c84f77-kube-api-access-tnvlx\") pod \"service-ca-5fffc97bdf-6b75b\" (UID: \"309f81fb-9b40-4053-a060-8672a7c84f77\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6b75b"
Oct 30 20:26:27.261251 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:27.261115    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-zmknv-22np8" event={"ID":"ee380d52-b71c-4839-8835-078d98a5e0ee","Type":"ContainerStarted","Data":"d39f3158be11fb9a929222afc0f19562ec8f1dc76da6a005cce4cb2cb62eeb39"}
Oct 30 20:26:27.261251 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:27.261173    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-zmknv-22np8" event={"ID":"ee380d52-b71c-4839-8835-078d98a5e0ee","Type":"ContainerStarted","Data":"3ce759cd9d7ce2099096ca884b37156c6e61e84eae795065d82f7c8b5d1a3d18"}
Oct 30 20:26:27.263766 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:27.263735    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pfvnn" event={"ID":"c9732ee5-c72e-44c6-9917-49cc9f81ffe0","Type":"ContainerStarted","Data":"6fcec7bd63880a568b866d7794a0a987a6a142b9641796d9607ac711466c00c5"}
Oct 30 20:26:27.263921 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:27.263853    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pfvnn"
Oct 30 20:26:27.265599 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:27.265575    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hbm7v" event={"ID":"50f563bf-3336-49c6-9708-041c41b41c53","Type":"ContainerStarted","Data":"9fe27797e37f4642def3994fea218e7ddb7c94d7f571fff9b06c0f5066ae7aba"}
Oct 30 20:26:27.302528 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:27.302500    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/309f81fb-9b40-4053-a060-8672a7c84f77-signing-key\") pod \"service-ca-5fffc97bdf-6b75b\" (UID: \"309f81fb-9b40-4053-a060-8672a7c84f77\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6b75b"
Oct 30 20:26:27.302688 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:27.302644    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/309f81fb-9b40-4053-a060-8672a7c84f77-signing-cabundle\") pod \"service-ca-5fffc97bdf-6b75b\" (UID: \"309f81fb-9b40-4053-a060-8672a7c84f77\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6b75b"
Oct 30 20:26:27.302755 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:27.302736    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnvlx\" (UniqueName: \"kubernetes.io/projected/309f81fb-9b40-4053-a060-8672a7c84f77-kube-api-access-tnvlx\") pod \"service-ca-5fffc97bdf-6b75b\" (UID: \"309f81fb-9b40-4053-a060-8672a7c84f77\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6b75b"
Oct 30 20:26:27.305709 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:27.305665    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/309f81fb-9b40-4053-a060-8672a7c84f77-signing-cabundle\") pod \"service-ca-5fffc97bdf-6b75b\" (UID: \"309f81fb-9b40-4053-a060-8672a7c84f77\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6b75b"
Oct 30 20:26:27.306730 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:27.306699    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/309f81fb-9b40-4053-a060-8672a7c84f77-signing-key\") pod \"service-ca-5fffc97bdf-6b75b\" (UID: \"309f81fb-9b40-4053-a060-8672a7c84f77\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6b75b"
Oct 30 20:26:27.380243 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:27.380145    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnvlx\" (UniqueName: \"kubernetes.io/projected/309f81fb-9b40-4053-a060-8672a7c84f77-kube-api-access-tnvlx\") pod \"service-ca-5fffc97bdf-6b75b\" (UID: \"309f81fb-9b40-4053-a060-8672a7c84f77\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6b75b"
Oct 30 20:26:27.396540 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:27.396511    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:27.445446 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:27.445408    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb"
Oct 30 20:26:27.445604 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:27.445513    2575 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb"
Oct 30 20:26:27.446126 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:27.446099    2575 scope.go:122] "RemoveContainer" containerID="b21ea8949c9de8f71605d238975ce27033db9c0ec4dff0ed0d0894a5dbcff357"
Oct 30 20:26:27.446386 ip-10-0-139-149 kubenswrapper[2575]: E1030 20:26:27.446364    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-l8qvb_openshift-console-operator(91fe9e06-0c30-4000-bddb-2ba89cd048cc)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb" podUID="91fe9e06-0c30-4000-bddb-2ba89cd048cc"
Oct 30 20:26:27.450732 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:27.450484    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-6b75b"
Oct 30 20:26:27.503169 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:27.502900    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-zmknv-22np8" podStartSLOduration=2.502879998 podStartE2EDuration="2.502879998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:27.379166366 +0000 UTC m=+41.950311447" watchObservedRunningTime="2026-10-30 20:26:27.502879998 +0000 UTC m=+42.074025079"
Oct 30 20:26:27.503354 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:27.503167    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pfvnn" podStartSLOduration=34.482887495 podStartE2EDuration="41.503159085s" totalImagesPullingTime="7.02027159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:25:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:27.502284794 +0000 UTC m=+42.073429874" watchObservedRunningTime="2026-10-30 20:26:27.503159085 +0000 UTC m=+42.074304165"
Oct 30 20:26:27.589238 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:27.589202    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:27.842007 ip-10-0-139-149 kubenswrapper[2575]: I1030 20:26:27.841944    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-6b75b"]
Oct 30 20:26:27.850036 ip-10-0-139-149 kubenswrapper[2575]: W1030 20:26:27.850006    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod309f81fb_9b40_4053_a060_8672a7c84f77.slice/crio-fc9e9f0e8e7de5163bf21bc8f42e80dd6f1dc238753973fd2fc6fac5adabb845 WatchSource:0}: Error finding container fc9e9f0e8e7de5163bf21bc8f42e80dd6f1dc238753973fd2fc6fac5adabb845: Status 404 returned error can't find the container with id fc9e9f0e8e7de5163bf21bc8f42e80dd6f1dc238753973fd2fc6fac5adabb845
Oct 30 20:26:28.028757 ip-10-0-139-149 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 20:26:28.037516 ip-10-0-139-149 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 20:26:28.037748 ip-10-0-139-149 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 20:26:28.037992 ip-10-0-139-149 systemd[1]: kubelet.service: Consumed 3.302s CPU time.
Oct 30 20:26:28.047609 ip-10-0-139-149 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 20:26:28.113341 ip-10-0-139-149 kubenswrapper[5071]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:26:28.113341 ip-10-0-139-149 kubenswrapper[5071]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 20:26:28.113341 ip-10-0-139-149 kubenswrapper[5071]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:26:28.113341 ip-10-0-139-149 kubenswrapper[5071]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:26:28.115553 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115478    5071 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:26:28.115553 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115552    5071 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:26:28.115553 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115556    5071 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:26:28.115553 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115559    5071 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:26:28.115725 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115562    5071 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:26:28.115725 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115565    5071 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:26:28.115725 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115568    5071 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:26:28.115725 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115571    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:26:28.115725 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115575    5071 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:26:28.115725 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115579    5071 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:26:28.115725 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115582    5071 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:26:28.115725 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115585    5071 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:26:28.115725 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115588    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:26:28.115725 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115590    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:26:28.115725 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115593    5071 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:26:28.115725 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115595    5071 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:26:28.115725 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115598    5071 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:26:28.115725 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115601    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:26:28.115725 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115603    5071 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:26:28.115725 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115605    5071 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:26:28.115725 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115608    5071 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:26:28.115725 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115611    5071 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:26:28.115725 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115613    5071 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:26:28.116199 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115616    5071 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:26:28.116199 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115619    5071 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:26:28.116199 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115622    5071 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:26:28.116199 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115629    5071 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:26:28.116199 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115633    5071 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:26:28.116199 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115635    5071 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:26:28.116199 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115638    5071 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:26:28.116199 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115640    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:26:28.116199 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115644    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:26:28.116199 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115647    5071 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:26:28.116199 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115650    5071 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:26:28.116199 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115652    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:26:28.116199 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115655    5071 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:26:28.116199 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115658    5071 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:26:28.116199 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115661    5071 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:26:28.116199 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115663    5071 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:26:28.116199 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115666    5071 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:26:28.116199 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115668    5071 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:26:28.116199 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115671    5071 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:26:28.116199 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115673    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:26:28.116676 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115676    5071 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:26:28.116676 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115679    5071 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:26:28.116676 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115682    5071 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:26:28.116676 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115684    5071 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:26:28.116676 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115687    5071 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:26:28.116676 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115689    5071 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:26:28.116676 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115692    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:26:28.116676 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115694    5071 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:26:28.116676 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115697    5071 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:26:28.116676 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115699    5071 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:26:28.116676 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115702    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:26:28.116676 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115705    5071 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:26:28.116676 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115707    5071 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:26:28.116676 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115709    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:26:28.116676 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115712    5071 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:26:28.116676 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115714    5071 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:26:28.116676 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115717    5071 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:26:28.116676 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115720    5071 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:26:28.116676 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115722    5071 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:26:28.116676 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115725    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:26:28.117191 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115728    5071 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:26:28.117191 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115731    5071 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:26:28.117191 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115733    5071 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:26:28.117191 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115736    5071 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:26:28.117191 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115739    5071 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:26:28.117191 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115742    5071 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:26:28.117191 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115748    5071 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:26:28.117191 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115751    5071 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:26:28.117191 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115753    5071 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:26:28.117191 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115756    5071 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:26:28.117191 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115759    5071 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:26:28.117191 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115761    5071 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:26:28.117191 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115764    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:26:28.117191 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115766    5071 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:26:28.117191 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115769    5071 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:26:28.117191 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115772    5071 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:26:28.117191 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115775    5071 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:26:28.117191 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115777    5071 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:26:28.117191 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115780    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:26:28.117191 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115782    5071 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:26:28.117654 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115785    5071 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:26:28.117654 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115787    5071 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:26:28.117654 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115790    5071 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:26:28.117654 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115792    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:26:28.117654 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115794    5071 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:26:28.117654 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115797    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:26:28.117654 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115799    5071 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:26:28.117654 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115802    5071 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:26:28.117654 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115804    5071 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:26:28.117654 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115807    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:26:28.117654 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115810    5071 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:26:28.117654 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115812    5071 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:26:28.117654 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115832    5071 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:26:28.117654 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115839    5071 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:26:28.117654 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115842    5071 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:26:28.117654 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115844    5071 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:26:28.117654 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115847    5071 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:26:28.117654 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115850    5071 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:26:28.117654 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115854    5071 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:26:28.118143 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115856    5071 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:26:28.118143 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115860    5071 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:26:28.118143 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115862    5071 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:26:28.118143 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.115865    5071 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:26:28.118143 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116636    5071 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:26:28.118143 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116644    5071 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:26:28.118143 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116649    5071 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:26:28.118143 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116652    5071 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:26:28.118143 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116655    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:26:28.118143 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116658    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:26:28.118143 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116661    5071 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:26:28.118143 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116664    5071 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:26:28.118143 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116667    5071 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:26:28.118143 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116669    5071 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:26:28.118143 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116672    5071 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:26:28.118143 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116674    5071 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:26:28.118143 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116677    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:26:28.118143 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116680    5071 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:26:28.118143 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116682    5071 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:26:28.118625 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116685    5071 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:26:28.118625 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116688    5071 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:26:28.118625 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116691    5071 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:26:28.118625 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116693    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:26:28.118625 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116695    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:26:28.118625 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116698    5071 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:26:28.118625 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116700    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:26:28.118625 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116704    5071 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:26:28.118625 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116707    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:26:28.118625 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116710    5071 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:26:28.118625 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116712    5071 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:26:28.118625 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116715    5071 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:26:28.118625 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116718    5071 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:26:28.118625 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116721    5071 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:26:28.118625 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116723    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:26:28.118625 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116726    5071 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:26:28.118625 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116728    5071 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:26:28.118625 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116730    5071 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:26:28.118625 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116733    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:26:28.119165 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116736    5071 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:26:28.119165 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116738    5071 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:26:28.119165 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116741    5071 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:26:28.119165 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116743    5071 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:26:28.119165 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116746    5071 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:26:28.119165 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116749    5071 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:26:28.119165 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116751    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:26:28.119165 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116754    5071 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:26:28.119165 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116756    5071 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:26:28.119165 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116759    5071 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:26:28.119165 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116761    5071 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:26:28.119165 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116763    5071 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:26:28.119165 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116766    5071 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:26:28.119165 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116768    5071 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:26:28.119165 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116771    5071 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:26:28.119165 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116773    5071 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:26:28.119165 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116775    5071 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:26:28.119165 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116778    5071 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:26:28.119165 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116781    5071 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:26:28.119624 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116784    5071 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:26:28.119624 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116787    5071 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:26:28.119624 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116789    5071 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:26:28.119624 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116792    5071 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:26:28.119624 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116795    5071 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:26:28.119624 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116797    5071 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:26:28.119624 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116800    5071 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:26:28.119624 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116802    5071 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:26:28.119624 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116804    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:26:28.119624 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116807    5071 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:26:28.119624 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116810    5071 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:26:28.119624 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116813    5071 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:26:28.119624 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116815    5071 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:26:28.119624 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116836    5071 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:26:28.119624 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116840    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:26:28.119624 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116842    5071 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:26:28.119624 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116845    5071 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:26:28.119624 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116847    5071 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:26:28.119624 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116856    5071 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:26:28.120106 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116859    5071 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:26:28.120106 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116862    5071 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:26:28.120106 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116865    5071 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:26:28.120106 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116867    5071 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:26:28.120106 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116870    5071 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:26:28.120106 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116878    5071 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:26:28.120106 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116882    5071 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:26:28.120106 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116884    5071 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:26:28.120106 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116887    5071 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:26:28.120106 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116890    5071 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:26:28.120106 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116892    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:26:28.120106 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116895    5071 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:26:28.120106 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116897    5071 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:26:28.120106 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116899    5071 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:26:28.120106 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116902    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:26:28.120106 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116905    5071 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:26:28.120106 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116907    5071 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:26:28.120106 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116909    5071 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:26:28.120106 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116912    5071 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:26:28.120106 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116914    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:26:28.120620 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116917    5071 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:26:28.120620 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116919    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:26:28.120620 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116922    5071 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:26:28.120620 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116924    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:26:28.120620 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116926    5071 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:26:28.120620 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116929    5071 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:26:28.120620 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116931    5071 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:26:28.120620 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116933    5071 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:26:28.120620 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116936    5071 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:26:28.120620 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116938    5071 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:26:28.120620 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116941    5071 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:26:28.120620 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116945    5071 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:26:28.120620 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116956    5071 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:26:28.120620 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.116960    5071 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:26:28.120620 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117518    5071 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:26:28.120620 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117525    5071 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:26:28.120620 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117529    5071 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:26:28.120620 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117531    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:26:28.120620 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117534    5071 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:26:28.121086 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117537    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:26:28.121086 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117540    5071 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:26:28.121086 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117542    5071 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:26:28.121086 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117545    5071 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:26:28.121086 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117547    5071 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:26:28.121086 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117549    5071 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:26:28.121086 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117553    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:26:28.121086 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117555    5071 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:26:28.121086 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117560    5071 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:26:28.121086 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117563    5071 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:26:28.121086 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117565    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:26:28.121086 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117568    5071 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:26:28.121086 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117570    5071 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:26:28.121086 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117573    5071 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:26:28.121086 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117575    5071 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:26:28.121086 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117578    5071 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:26:28.121086 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117580    5071 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:26:28.121086 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117583    5071 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:26:28.121086 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117585    5071 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:26:28.121546 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117587    5071 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:26:28.121546 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117590    5071 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:26:28.121546 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117592    5071 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:26:28.121546 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117595    5071 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:26:28.121546 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117597    5071 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:26:28.121546 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117600    5071 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:26:28.121546 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117608    5071 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:26:28.121546 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117611    5071 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:26:28.121546 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117614    5071 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:26:28.121546 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117616    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:26:28.121546 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117619    5071 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:26:28.121546 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117622    5071 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:26:28.121546 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117624    5071 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:26:28.121546 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117627    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:26:28.121546 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117629    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:26:28.121546 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117632    5071 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:26:28.121546 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117634    5071 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:26:28.121546 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117636    5071 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:26:28.121546 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117639    5071 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:26:28.121546 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117641    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:26:28.122037 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117644    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:26:28.122037 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117648    5071 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:26:28.122037 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117650    5071 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:26:28.122037 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117653    5071 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:26:28.122037 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117655    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:26:28.122037 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117658    5071 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:26:28.122037 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117661    5071 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:26:28.122037 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117663    5071 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:26:28.122037 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117666    5071 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:26:28.122037 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117668    5071 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:26:28.122037 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117671    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:26:28.122037 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117674    5071 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:26:28.122037 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117676    5071 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:26:28.122037 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117679    5071 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:26:28.122037 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117681    5071 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:26:28.122037 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117684    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:26:28.122037 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117687    5071 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:26:28.122037 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117689    5071 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:26:28.122037 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117692    5071 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:26:28.122775 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117704    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:26:28.122775 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117706    5071 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:26:28.122775 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117709    5071 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:26:28.122775 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117712    5071 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:26:28.122775 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117714    5071 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:26:28.122775 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117716    5071 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:26:28.122775 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117719    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:26:28.122775 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117721    5071 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:26:28.122775 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117724    5071 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:26:28.122775 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117726    5071 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:26:28.122775 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117729    5071 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:26:28.122775 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117732    5071 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:26:28.122775 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117734    5071 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:26:28.122775 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117737    5071 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:26:28.122775 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117741    5071 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:26:28.122775 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117744    5071 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:26:28.122775 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117746    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:26:28.122775 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117748    5071 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:26:28.122775 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117751    5071 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:26:28.122775 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117753    5071 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:26:28.123398 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117756    5071 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:26:28.123398 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117758    5071 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:26:28.123398 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117761    5071 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:26:28.123398 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117763    5071 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:26:28.123398 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117765    5071 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:26:28.123398 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117769    5071 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:26:28.123398 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117772    5071 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:26:28.123398 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117775    5071 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:26:28.123398 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117778    5071 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:26:28.123398 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117780    5071 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:26:28.123398 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117782    5071 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:26:28.123398 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117785    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:26:28.123398 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117787    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:26:28.123398 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117795    5071 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:26:28.123398 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117798    5071 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:26:28.123398 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117801    5071 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:26:28.123398 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117803    5071 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:26:28.123398 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117806    5071 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:26:28.123398 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117809    5071 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:26:28.123891 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117811    5071 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:26:28.123891 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117813    5071 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:26:28.123891 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117834    5071 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:26:28.123891 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.117837    5071 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:26:28.123891 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.117939    5071 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 20:26:28.123891 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.117955    5071 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 20:26:28.123891 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.117963    5071 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 20:26:28.123891 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.117970    5071 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 20:26:28.123891 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.117974    5071 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 20:26:28.123891 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.117977    5071 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 20:26:28.123891 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.117981    5071 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 20:26:28.123891 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.117985    5071 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 20:26:28.123891 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.117988    5071 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 20:26:28.123891 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.117991    5071 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 20:26:28.123891 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.117994    5071 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 20:26:28.123891 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.117998    5071 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 20:26:28.123891 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118001    5071 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 20:26:28.123891 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118004    5071 flags.go:64] FLAG: --cgroup-root=""
Oct 30 20:26:28.123891 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118006    5071 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 20:26:28.123891 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118009    5071 flags.go:64] FLAG: --client-ca-file=""
Oct 30 20:26:28.123891 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118012    5071 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 20:26:28.123891 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118017    5071 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 20:26:28.123891 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118024    5071 flags.go:64] FLAG: --cluster-domain=""
Oct 30 20:26:28.123891 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118027    5071 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 20:26:28.124451 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118030    5071 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 20:26:28.124451 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118033    5071 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 20:26:28.124451 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118036    5071 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 20:26:28.124451 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118045    5071 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 20:26:28.124451 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118048    5071 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 20:26:28.124451 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118051    5071 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 20:26:28.124451 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118055    5071 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 20:26:28.124451 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118058    5071 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 20:26:28.124451 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118061    5071 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 20:26:28.124451 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118064    5071 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 20:26:28.124451 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118067    5071 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 20:26:28.124451 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118070    5071 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 20:26:28.124451 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118074    5071 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 20:26:28.124451 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118077    5071 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 20:26:28.124451 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118080    5071 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 20:26:28.124451 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118085    5071 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 20:26:28.124451 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118088    5071 flags.go:64] FLAG: --enable-server="true"
Oct 30 20:26:28.124451 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118091    5071 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 20:26:28.124451 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118097    5071 flags.go:64] FLAG: --event-burst="100"
Oct 30 20:26:28.124451 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118101    5071 flags.go:64] FLAG: --event-qps="50"
Oct 30 20:26:28.124451 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118104    5071 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 20:26:28.124451 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118107    5071 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 20:26:28.124451 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118110    5071 flags.go:64] FLAG: --eviction-hard=""
Oct 30 20:26:28.124451 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118114    5071 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 20:26:28.125043 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118117    5071 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 20:26:28.125043 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118120    5071 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 20:26:28.125043 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118124    5071 flags.go:64] FLAG: --eviction-soft=""
Oct 30 20:26:28.125043 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118126    5071 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 20:26:28.125043 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118129    5071 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 20:26:28.125043 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118133    5071 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 20:26:28.125043 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118136    5071 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 20:26:28.125043 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118139    5071 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 20:26:28.125043 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118142    5071 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 20:26:28.125043 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118145    5071 flags.go:64] FLAG: --feature-gates=""
Oct 30 20:26:28.125043 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118149    5071 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 20:26:28.125043 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118152    5071 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 20:26:28.125043 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118163    5071 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 20:26:28.125043 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118167    5071 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 20:26:28.125043 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118170    5071 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 20:26:28.125043 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118173    5071 flags.go:64] FLAG: --help="false"
Oct 30 20:26:28.125043 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118176    5071 flags.go:64] FLAG: --hostname-override="ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:26:28.125043 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118179    5071 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 20:26:28.125043 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118182    5071 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 20:26:28.125043 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118185    5071 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 20:26:28.125043 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118188    5071 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 20:26:28.125043 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118191    5071 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 20:26:28.125043 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118194    5071 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 20:26:28.125579 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118198    5071 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 20:26:28.125579 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118201    5071 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 20:26:28.125579 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118204    5071 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 20:26:28.125579 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118207    5071 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 20:26:28.125579 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118210    5071 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 20:26:28.125579 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118213    5071 flags.go:64] FLAG: --kube-reserved=""
Oct 30 20:26:28.125579 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118216    5071 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 20:26:28.125579 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118218    5071 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 20:26:28.125579 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118221    5071 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 20:26:28.125579 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118224    5071 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 20:26:28.125579 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118227    5071 flags.go:64] FLAG: --lock-file=""
Oct 30 20:26:28.125579 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118230    5071 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 20:26:28.125579 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118233    5071 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 20:26:28.125579 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118236    5071 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 20:26:28.125579 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118243    5071 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 20:26:28.125579 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118246    5071 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 20:26:28.125579 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118249    5071 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 20:26:28.125579 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118252    5071 flags.go:64] FLAG: --logging-format="text"
Oct 30 20:26:28.125579 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118255    5071 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 20:26:28.125579 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118258    5071 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 20:26:28.125579 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118261    5071 flags.go:64] FLAG: --manifest-url=""
Oct 30 20:26:28.125579 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118263    5071 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 20:26:28.125579 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118273    5071 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 20:26:28.125579 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118277    5071 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 20:26:28.125579 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118281    5071 flags.go:64] FLAG: --max-pods="110"
Oct 30 20:26:28.126514 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118283    5071 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 20:26:28.126514 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118286    5071 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 20:26:28.126514 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118289    5071 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 20:26:28.126514 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118292    5071 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 20:26:28.126514 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118295    5071 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 20:26:28.126514 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118298    5071 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 20:26:28.126514 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118301    5071 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 20:26:28.126514 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118309    5071 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 20:26:28.126514 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118312    5071 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 20:26:28.126514 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118315    5071 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 20:26:28.126514 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118319    5071 flags.go:64] FLAG: --pod-cidr=""
Oct 30 20:26:28.126514 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118321    5071 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 20:26:28.126514 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118324    5071 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 20:26:28.126514 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118327    5071 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 20:26:28.126514 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118330    5071 flags.go:64] FLAG: --port="10250"
Oct 30 20:26:28.126514 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118333    5071 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 20:26:28.126514 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118335    5071 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-04bcadb1c15a1ecbe"
Oct 30 20:26:28.126514 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118338    5071 flags.go:64] FLAG: --qos-reserved=""
Oct 30 20:26:28.126514 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118341    5071 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 20:26:28.126514 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118344    5071 flags.go:64] FLAG: --register-node="true"
Oct 30 20:26:28.126514 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118347    5071 flags.go:64] FLAG: --register-with-taints=""
Oct 30 20:26:28.126514 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118352    5071 flags.go:64] FLAG: --registry-burst="10"
Oct 30 20:26:28.126514 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118355    5071 flags.go:64] FLAG: --registry-qps="5"
Oct 30 20:26:28.126514 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118358    5071 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 20:26:28.126514 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118361    5071 flags.go:64] FLAG: --reserved-memory=""
Oct 30 20:26:28.126514 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118365    5071 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 20:26:28.127218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118368    5071 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 20:26:28.127218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118371    5071 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 20:26:28.127218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118373    5071 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 20:26:28.127218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118376    5071 flags.go:64] FLAG: --runonce="false"
Oct 30 20:26:28.127218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118385    5071 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 20:26:28.127218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118389    5071 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 20:26:28.127218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118392    5071 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 20:26:28.127218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118395    5071 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 20:26:28.127218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118397    5071 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 20:26:28.127218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118401    5071 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 20:26:28.127218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118404    5071 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 20:26:28.127218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118406    5071 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 20:26:28.127218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118409    5071 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 20:26:28.127218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118413    5071 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 20:26:28.127218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118416    5071 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 20:26:28.127218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118419    5071 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 20:26:28.127218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118422    5071 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 20:26:28.127218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118425    5071 flags.go:64] FLAG: --system-cgroups=""
Oct 30 20:26:28.127218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118427    5071 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 20:26:28.127218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118433    5071 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 20:26:28.127218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118435    5071 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 20:26:28.127218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118438    5071 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 20:26:28.127218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118445    5071 flags.go:64] FLAG: --tls-min-version=""
Oct 30 20:26:28.127218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118448    5071 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 20:26:28.127218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118450    5071 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 20:26:28.127839 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118453    5071 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 20:26:28.127839 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118456    5071 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 20:26:28.127839 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118460    5071 flags.go:64] FLAG: --v="2"
Oct 30 20:26:28.127839 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118465    5071 flags.go:64] FLAG: --version="false"
Oct 30 20:26:28.127839 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118469    5071 flags.go:64] FLAG: --vmodule=""
Oct 30 20:26:28.127839 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118472    5071 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 20:26:28.127839 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.118475    5071 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 20:26:28.127839 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118694    5071 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:26:28.127839 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118699    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:26:28.127839 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118702    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:26:28.127839 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118705    5071 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:26:28.127839 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118708    5071 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:26:28.127839 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118716    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:26:28.127839 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118719    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:26:28.127839 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118721    5071 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:26:28.127839 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118724    5071 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:26:28.127839 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118726    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:26:28.127839 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118729    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:26:28.127839 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118731    5071 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:26:28.127839 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118734    5071 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:26:28.127839 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118737    5071 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:26:28.128387 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118740    5071 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:26:28.128387 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118742    5071 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:26:28.128387 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118747    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:26:28.128387 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118750    5071 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:26:28.128387 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118754    5071 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:26:28.128387 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118758    5071 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:26:28.128387 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118760    5071 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:26:28.128387 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118763    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:26:28.128387 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118765    5071 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:26:28.128387 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118768    5071 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:26:28.128387 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118770    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:26:28.128387 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118773    5071 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:26:28.128387 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118775    5071 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:26:28.128387 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118778    5071 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:26:28.128387 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118781    5071 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:26:28.128387 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118783    5071 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:26:28.128387 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118786    5071 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:26:28.128387 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118788    5071 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:26:28.128387 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118791    5071 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:26:28.128845 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118793    5071 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:26:28.128845 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118795    5071 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:26:28.128845 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118798    5071 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:26:28.128845 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118800    5071 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:26:28.128845 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118802    5071 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:26:28.128845 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118806    5071 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:26:28.128845 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118809    5071 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:26:28.128845 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118811    5071 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:26:28.128845 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118813    5071 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:26:28.128845 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118816    5071 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:26:28.128845 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118838    5071 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:26:28.128845 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118842    5071 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:26:28.128845 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118846    5071 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:26:28.128845 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118850    5071 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:26:28.128845 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118853    5071 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:26:28.128845 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118855    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:26:28.128845 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118858    5071 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:26:28.128845 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118860    5071 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:26:28.128845 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118863    5071 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:26:28.128845 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118865    5071 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:26:28.129466 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118868    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:26:28.129466 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118870    5071 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:26:28.129466 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118873    5071 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:26:28.129466 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118875    5071 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:26:28.129466 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118877    5071 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:26:28.129466 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118880    5071 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:26:28.129466 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118882    5071 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:26:28.129466 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118885    5071 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:26:28.129466 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118887    5071 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:26:28.129466 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118890    5071 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:26:28.129466 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118893    5071 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:26:28.129466 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118895    5071 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:26:28.129466 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118898    5071 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:26:28.129466 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118900    5071 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:26:28.129466 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118903    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:26:28.129466 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118905    5071 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:26:28.129466 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118908    5071 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:26:28.129466 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118910    5071 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:26:28.129466 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118913    5071 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:26:28.130087 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118916    5071 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:26:28.130087 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118919    5071 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:26:28.130087 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118921    5071 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:26:28.130087 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118924    5071 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:26:28.130087 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118926    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:26:28.130087 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118929    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:26:28.130087 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118931    5071 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:26:28.130087 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118933    5071 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:26:28.130087 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118936    5071 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:26:28.130087 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118938    5071 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:26:28.130087 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118941    5071 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:26:28.130087 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118943    5071 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:26:28.130087 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118945    5071 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:26:28.130087 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118948    5071 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:26:28.130087 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118950    5071 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:26:28.130087 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118953    5071 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:26:28.130087 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118955    5071 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:26:28.130087 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118957    5071 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:26:28.130087 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118960    5071 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:26:28.130087 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118963    5071 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:26:28.130719 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118965    5071 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:26:28.130719 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118968    5071 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:26:28.130719 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118970    5071 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:26:28.130719 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118973    5071 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:26:28.130719 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118976    5071 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:26:28.130719 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118980    5071 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:26:28.130719 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118983    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:26:28.130719 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118986    5071 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:26:28.130719 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118988    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:26:28.130719 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118992    5071 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:26:28.130719 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118994    5071 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:26:28.130719 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.118997    5071 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:26:28.130719 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119000    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:26:28.130719 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119002    5071 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:26:28.130719 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.119028    5071 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:26:28.130719 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119070    5071 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:26:28.131216 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119074    5071 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:26:28.131216 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119077    5071 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:26:28.131216 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119079    5071 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:26:28.131216 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119082    5071 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:26:28.131216 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119084    5071 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:26:28.131216 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119087    5071 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:26:28.131216 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119090    5071 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:26:28.131216 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119093    5071 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:26:28.131216 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119096    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:26:28.131216 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119101    5071 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:26:28.131216 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119103    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:26:28.131216 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119106    5071 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:26:28.131216 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119109    5071 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:26:28.131216 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119111    5071 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:26:28.131216 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119114    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:26:28.131216 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119116    5071 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:26:28.131216 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119119    5071 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:26:28.131216 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119121    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:26:28.131216 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119124    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:26:28.131216 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119126    5071 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:26:28.131705 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119129    5071 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:26:28.131705 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119131    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:26:28.131705 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119133    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:26:28.131705 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119136    5071 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:26:28.131705 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119138    5071 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:26:28.131705 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119141    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:26:28.131705 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119143    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:26:28.131705 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119146    5071 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:26:28.131705 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119148    5071 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:26:28.131705 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119151    5071 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:26:28.131705 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119154    5071 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:26:28.131705 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119157    5071 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:26:28.131705 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119159    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:26:28.131705 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119161    5071 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:26:28.131705 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119165    5071 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:26:28.131705 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119170    5071 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:26:28.131705 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119173    5071 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:26:28.131705 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119176    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:26:28.131705 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119178    5071 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:26:28.132362 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119181    5071 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:26:28.132362 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119184    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:26:28.132362 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119194    5071 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:26:28.132362 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119198    5071 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:26:28.132362 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119201    5071 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:26:28.132362 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119203    5071 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:26:28.132362 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119206    5071 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:26:28.132362 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119209    5071 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:26:28.132362 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119212    5071 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:26:28.132362 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119214    5071 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:26:28.132362 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119216    5071 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:26:28.132362 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119219    5071 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:26:28.132362 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119221    5071 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:26:28.132362 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119224    5071 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:26:28.132362 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119226    5071 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:26:28.132362 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119229    5071 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:26:28.132362 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119231    5071 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:26:28.132362 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119234    5071 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:26:28.132362 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119236    5071 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:26:28.132362 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119239    5071 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:26:28.132960 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119241    5071 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:26:28.132960 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119243    5071 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:26:28.132960 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119246    5071 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:26:28.132960 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119254    5071 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:26:28.132960 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119256    5071 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:26:28.132960 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119259    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:26:28.132960 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119261    5071 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:26:28.132960 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119264    5071 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:26:28.132960 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119267    5071 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:26:28.132960 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119269    5071 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:26:28.132960 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119271    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:26:28.132960 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119274    5071 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:26:28.132960 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119276    5071 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:26:28.132960 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119279    5071 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:26:28.132960 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119283    5071 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:26:28.132960 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119287    5071 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:26:28.132960 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119289    5071 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:26:28.132960 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119292    5071 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:26:28.132960 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119294    5071 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:26:28.133417 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119297    5071 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:26:28.133417 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119299    5071 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:26:28.133417 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119302    5071 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:26:28.133417 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119304    5071 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:26:28.133417 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119307    5071 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:26:28.133417 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119309    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:26:28.133417 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119312    5071 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:26:28.133417 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119314    5071 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:26:28.133417 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119317    5071 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:26:28.133417 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119319    5071 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:26:28.133417 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119322    5071 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:26:28.133417 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119324    5071 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:26:28.133417 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119327    5071 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:26:28.133417 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119329    5071 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:26:28.133417 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119332    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:26:28.133417 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119335    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:26:28.133417 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119338    5071 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:26:28.133417 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119347    5071 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:26:28.133417 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119350    5071 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:26:28.133417 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119352    5071 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:26:28.133924 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119355    5071 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:26:28.133924 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119357    5071 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:26:28.133924 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119359    5071 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:26:28.133924 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119362    5071 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:26:28.133924 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119364    5071 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:26:28.133924 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119367    5071 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:26:28.133924 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.119370    5071 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:26:28.133924 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.124343    5071 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 20:26:28.133924 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.124360    5071 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 20:26:28.133924 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.124380    5071 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 20:26:28.133924 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.124384    5071 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 20:26:28.133924 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124444    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:26:28.133924 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124449    5071 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:26:28.133924 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124453    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:26:28.133924 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124456    5071 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:26:28.133924 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124459    5071 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:26:28.133924 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124462    5071 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:26:28.133924 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124465    5071 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:26:28.133924 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124468    5071 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:26:28.133924 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124470    5071 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:26:28.134411 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124473    5071 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:26:28.134411 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124475    5071 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:26:28.134411 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124478    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:26:28.134411 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124481    5071 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:26:28.134411 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124483    5071 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:26:28.134411 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124485    5071 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:26:28.134411 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124488    5071 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:26:28.134411 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124490    5071 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:26:28.134411 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124493    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:26:28.134411 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124495    5071 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:26:28.134411 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124498    5071 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:26:28.134411 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124500    5071 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:26:28.134411 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124503    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:26:28.134411 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124505    5071 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:26:28.134411 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124508    5071 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:26:28.134411 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124511    5071 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:26:28.134411 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124513    5071 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:26:28.134411 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124516    5071 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:26:28.134411 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124519    5071 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:26:28.134411 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124521    5071 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:26:28.135070 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124524    5071 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:26:28.135070 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124526    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:26:28.135070 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124529    5071 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:26:28.135070 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124532    5071 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:26:28.135070 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124534    5071 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:26:28.135070 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124536    5071 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:26:28.135070 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124539    5071 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:26:28.135070 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124542    5071 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:26:28.135070 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124544    5071 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:26:28.135070 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124547    5071 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:26:28.135070 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124549    5071 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:26:28.135070 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124552    5071 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:26:28.135070 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124554    5071 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:26:28.135070 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124556    5071 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:26:28.135070 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124559    5071 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:26:28.135070 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124562    5071 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:26:28.135070 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124565    5071 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:26:28.135070 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124568    5071 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:26:28.135070 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124570    5071 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:26:28.135070 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124573    5071 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:26:28.135593 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124575    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:26:28.135593 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124577    5071 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:26:28.135593 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124580    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:26:28.135593 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124582    5071 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:26:28.135593 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124586    5071 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:26:28.135593 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124588    5071 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:26:28.135593 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124591    5071 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:26:28.135593 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124593    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:26:28.135593 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124596    5071 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:26:28.135593 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124598    5071 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:26:28.135593 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124601    5071 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:26:28.135593 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124605    5071 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:26:28.135593 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124608    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:26:28.135593 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124612    5071 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:26:28.135593 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124615    5071 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:26:28.135593 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124618    5071 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:26:28.135593 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124620    5071 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:26:28.135593 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124623    5071 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:26:28.135593 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124625    5071 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:26:28.136203 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124628    5071 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:26:28.136203 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124630    5071 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:26:28.136203 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124633    5071 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:26:28.136203 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124635    5071 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:26:28.136203 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124637    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:26:28.136203 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124640    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:26:28.136203 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124642    5071 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:26:28.136203 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124645    5071 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:26:28.136203 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124647    5071 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:26:28.136203 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124650    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:26:28.136203 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124652    5071 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:26:28.136203 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124655    5071 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:26:28.136203 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124658    5071 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:26:28.136203 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124661    5071 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:26:28.136203 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124663    5071 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:26:28.136203 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124666    5071 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:26:28.136203 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124668    5071 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:26:28.136203 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124671    5071 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:26:28.136203 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124675    5071 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:26:28.136659 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124678    5071 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:26:28.136659 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124681    5071 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:26:28.136659 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124685    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:26:28.136659 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124687    5071 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:26:28.136659 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124690    5071 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:26:28.136659 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124692    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:26:28.136659 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124695    5071 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:26:28.136659 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124697    5071 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:26:28.136659 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124700    5071 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:26:28.136659 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124702    5071 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:26:28.136659 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124705    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:26:28.136659 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124707    5071 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:26:28.136659 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124710    5071 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:26:28.136659 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124712    5071 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:26:28.136659 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124715    5071 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:26:28.136659 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124717    5071 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:26:28.136659 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124719    5071 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:26:28.136659 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124722    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:26:28.136659 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124724    5071 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:26:28.137123 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.124751    5071 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:26:28.137123 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124923    5071 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:26:28.137123 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124929    5071 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:26:28.137123 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124933    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:26:28.137123 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124935    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:26:28.137123 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124938    5071 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:26:28.137123 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124941    5071 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:26:28.137123 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124944    5071 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:26:28.137123 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124947    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:26:28.137123 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124949    5071 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:26:28.137123 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124952    5071 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:26:28.137123 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124954    5071 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:26:28.137123 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124957    5071 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:26:28.137123 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124959    5071 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:26:28.137123 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124962    5071 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:26:28.137123 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124964    5071 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:26:28.137496 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124967    5071 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:26:28.137496 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124969    5071 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:26:28.137496 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124971    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:26:28.137496 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124974    5071 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:26:28.137496 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124976    5071 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:26:28.137496 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124978    5071 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:26:28.137496 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124981    5071 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:26:28.137496 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124983    5071 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:26:28.137496 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124986    5071 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:26:28.137496 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124988    5071 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:26:28.137496 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124990    5071 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:26:28.137496 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124993    5071 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:26:28.137496 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124995    5071 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:26:28.137496 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.124997    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:26:28.137496 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125000    5071 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:26:28.137496 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125003    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:26:28.137496 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125005    5071 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:26:28.137496 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125008    5071 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:26:28.137496 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125012    5071 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:26:28.138045 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125016    5071 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:26:28.138045 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125019    5071 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:26:28.138045 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125022    5071 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:26:28.138045 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125024    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:26:28.138045 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125028    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:26:28.138045 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125030    5071 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:26:28.138045 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125033    5071 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:26:28.138045 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125036    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:26:28.138045 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125038    5071 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:26:28.138045 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125041    5071 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:26:28.138045 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125043    5071 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:26:28.138045 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125045    5071 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:26:28.138045 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125048    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:26:28.138045 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125051    5071 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:26:28.138045 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125054    5071 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:26:28.138045 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125056    5071 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:26:28.138045 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125059    5071 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:26:28.138045 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125061    5071 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:26:28.138045 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125063    5071 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:26:28.138521 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125066    5071 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:26:28.138521 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125068    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:26:28.138521 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125071    5071 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:26:28.138521 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125073    5071 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:26:28.138521 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125075    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:26:28.138521 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125078    5071 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:26:28.138521 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125080    5071 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:26:28.138521 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125082    5071 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:26:28.138521 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125085    5071 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:26:28.138521 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125090    5071 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:26:28.138521 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125092    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:26:28.138521 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125096    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:26:28.138521 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125098    5071 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:26:28.138521 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125101    5071 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:26:28.138521 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125103    5071 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:26:28.138521 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125106    5071 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:26:28.138521 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125108    5071 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:26:28.138521 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125111    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:26:28.138521 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125115    5071 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:26:28.138521 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125117    5071 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:26:28.139026 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125120    5071 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:26:28.139026 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125122    5071 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:26:28.139026 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125125    5071 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:26:28.139026 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125127    5071 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:26:28.139026 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125130    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:26:28.139026 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125132    5071 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:26:28.139026 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125135    5071 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:26:28.139026 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125137    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:26:28.139026 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125139    5071 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:26:28.139026 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125142    5071 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:26:28.139026 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125144    5071 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:26:28.139026 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125147    5071 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:26:28.139026 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125149    5071 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:26:28.139026 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125152    5071 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:26:28.139026 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125154    5071 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:26:28.139026 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125157    5071 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:26:28.139026 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125160    5071 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:26:28.139026 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125162    5071 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:26:28.139026 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125165    5071 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:26:28.139026 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125167    5071 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:26:28.139494 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125169    5071 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:26:28.139494 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125173    5071 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:26:28.139494 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125176    5071 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:26:28.139494 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125178    5071 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:26:28.139494 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125181    5071 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:26:28.139494 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125183    5071 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:26:28.139494 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125187    5071 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:26:28.139494 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125190    5071 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:26:28.139494 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125194    5071 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:26:28.139494 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125197    5071 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:26:28.139494 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125199    5071 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:26:28.139494 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125202    5071 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:26:28.139494 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125205    5071 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:26:28.139494 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.125229    5071 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:26:28.139494 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125260    5071 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:26:28.139494 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125264    5071 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:26:28.140036 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125267    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:26:28.140036 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125270    5071 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:26:28.140036 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125272    5071 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:26:28.140036 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125275    5071 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:26:28.140036 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125278    5071 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:26:28.140036 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125280    5071 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:26:28.140036 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125283    5071 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:26:28.140036 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125285    5071 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:26:28.140036 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125287    5071 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:26:28.140036 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125290    5071 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:26:28.140036 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125292    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:26:28.140036 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125295    5071 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:26:28.140036 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125297    5071 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:26:28.140036 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125300    5071 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:26:28.140036 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125302    5071 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:26:28.140036 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125305    5071 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:26:28.140036 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125307    5071 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:26:28.140036 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125309    5071 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:26:28.140036 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125313    5071 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:26:28.140486 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125315    5071 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:26:28.140486 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125318    5071 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:26:28.140486 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125321    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:26:28.140486 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125324    5071 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:26:28.140486 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125326    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:26:28.140486 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125328    5071 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:26:28.140486 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125331    5071 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:26:28.140486 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125335    5071 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:26:28.140486 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125338    5071 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:26:28.140486 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125342    5071 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:26:28.140486 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125345    5071 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:26:28.140486 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125348    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:26:28.140486 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125350    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:26:28.140486 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125353    5071 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:26:28.140486 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125355    5071 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:26:28.140486 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125358    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:26:28.140486 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125360    5071 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:26:28.140486 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125362    5071 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:26:28.140486 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125365    5071 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:26:28.141190 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125367    5071 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:26:28.141190 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125370    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:26:28.141190 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125372    5071 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:26:28.141190 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125375    5071 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:26:28.141190 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125377    5071 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:26:28.141190 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125380    5071 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:26:28.141190 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125384    5071 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:26:28.141190 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125387    5071 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:26:28.141190 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125389    5071 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:26:28.141190 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125392    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:26:28.141190 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125394    5071 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:26:28.141190 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125396    5071 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:26:28.141190 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125400    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:26:28.141190 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125403    5071 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:26:28.141190 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125405    5071 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:26:28.141190 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125408    5071 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:26:28.141190 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125410    5071 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:26:28.141190 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125413    5071 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:26:28.141190 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125415    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:26:28.141932 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125418    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:26:28.141932 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125420    5071 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:26:28.141932 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125423    5071 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:26:28.141932 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125425    5071 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:26:28.141932 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125428    5071 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:26:28.141932 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125431    5071 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:26:28.141932 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125433    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:26:28.141932 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125436    5071 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:26:28.141932 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125438    5071 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:26:28.141932 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125440    5071 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:26:28.141932 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125443    5071 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:26:28.141932 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125445    5071 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:26:28.141932 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125448    5071 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:26:28.141932 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125450    5071 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:26:28.141932 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125453    5071 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:26:28.141932 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125455    5071 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:26:28.141932 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125458    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:26:28.141932 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125460    5071 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:26:28.141932 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125462    5071 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:26:28.141932 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125466    5071 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:26:28.142470 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125468    5071 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:26:28.142470 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125471    5071 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:26:28.142470 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125473    5071 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:26:28.142470 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125476    5071 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:26:28.142470 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125478    5071 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:26:28.142470 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125485    5071 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:26:28.142470 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125487    5071 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:26:28.142470 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125490    5071 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:26:28.142470 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125492    5071 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:26:28.142470 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125495    5071 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:26:28.142470 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125497    5071 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:26:28.142470 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125500    5071 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:26:28.142470 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125502    5071 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:26:28.142470 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125504    5071 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:26:28.142470 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125507    5071 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:26:28.142470 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125511    5071 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:26:28.142470 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125514    5071 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:26:28.142470 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125519    5071 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:26:28.142470 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125522    5071 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:26:28.143081 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125524    5071 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:26:28.143081 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125527    5071 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:26:28.143081 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125529    5071 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:26:28.143081 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125532    5071 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:26:28.143081 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125535    5071 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:26:28.143081 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125538    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:26:28.143081 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125541    5071 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:26:28.143081 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:28.125543    5071 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:26:28.143081 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.125607    5071 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 20:26:28.143081 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.126606    5071 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 20:26:28.143081 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.126670    5071 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 20:26:28.143081 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.126886    5071 server.go:1014] "Starting client certificate rotation"
Oct 30 20:26:28.143081 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.127079    5071 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 20:26:28.143081 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.127131    5071 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 20:20:46 +0000 UTC" deadline="2028-05-26 04:01:24.20835298 +0000 UTC"
Oct 30 20:26:28.143081 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.127174    5071 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13759h34m56.081181277s"
Oct 30 20:26:28.143081 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.127382    5071 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 20:26:28.143473 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.128506    5071 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 20:26:28.143473 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.130058    5071 log.go:25] "Validated CRI v1 runtime API"
Oct 30 20:26:28.143473 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.131804    5071 log.go:25] "Validated CRI v1 image API"
Oct 30 20:26:28.143473 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.132748    5071 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 20:26:28.143473 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.134738    5071 fs.go:135] Filesystem UUIDs: map[73df03d4-9128-4efa-8ca8-0c4e682c0a29:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8a73f35e-5ef0-43e1-88f3-50fa93954dcd:/dev/nvme0n1p3]
Oct 30 20:26:28.143632 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.134763    5071 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0c839da3408e96750313810ca2107e90b21c719237b93eddfac858956709cad8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0c839da3408e96750313810ca2107e90b21c719237b93eddfac858956709cad8/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1a20f9bbd988db5a16fa334184144ae38c28e82d8703967417cc8b100e7b2031/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1a20f9bbd988db5a16fa334184144ae38c28e82d8703967417cc8b100e7b2031/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/24bfadd2814fc8daad30a6323ebc6fe65849fec0b2d2c1689868fc5fa2e0ada1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/24bfadd2814fc8daad30a6323ebc6fe65849fec0b2d2c1689868fc5fa2e0ada1/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/30316f990be28a3c33df9b2c59fd5df78eb099b0430afed2ca8b4801cd011ff7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/30316f990be28a3c33df9b2c59fd5df78eb099b0430afed2ca8b4801cd011ff7/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/38fc45f90c3b66e3c7d6b46a31d34de20e8ba9732b4a3b5841ea69328045fbc5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/38fc45f90c3b66e3c7d6b46a31d34de20e8ba9732b4a3b5841ea69328045fbc5/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3ce759cd9d7ce2099096ca884b37156c6e61e84eae795065d82f7c8b5d1a3d18/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3ce759cd9d7ce2099096ca884b37156c6e61e84eae795065d82f7c8b5d1a3d18/userdata/shm major:0 minor:338 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/50c63a4bd7251e656a4eaea02805ff737213b40bee52f7fe6c2b4e73e32e21a8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/50c63a4bd7251e656a4eaea02805ff737213b40bee52f7fe6c2b4e73e32e21a8/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/74f6fb01bca34a911a5b4074647d79ef03b18d23b55e713aca7d8b452344a7ae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/74f6fb01bca34a911a5b4074647d79ef03b18d23b55e713aca7d8b452344a7ae/userdata/shm major:0 minor:220 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8887ae7a21f88fa23debe43d42535d7b88646ed33ceb9eab7ef9c33c38edc394/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8887ae7a21f88fa23debe43d42535d7b88646ed33ceb9eab7ef9c33c38edc394/userdata/shm major:0 minor:333 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/893a77ea05e781f02d98656c4e2afe76248c31558482172f870102e7960d2962/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/893a77ea05e781f02d98656c4e2afe76248c31558482172f870102e7960d2962/userdata/shm major:0 minor:63 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8a1271576ec3b8c9517dd5d184a1f8c2c1b2c19190b24ba6ef7ae3a9cf805ea4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8a1271576ec3b8c9517dd5d184a1f8c2c1b2c19190b24ba6ef7ae3a9cf805ea4/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f5546fcc9c0952128d607267dd092c06ca09b46cec6046bab3a3c738e3040d4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f5546fcc9c0952128d607267dd092c06ca09b46cec6046bab3a3c738e3040d4/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f982210565e5738a437f8b5a6bea6e6a28dfdbfe4fadaa5e4828dde56b8f30c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f982210565e5738a437f8b5a6bea6e6a28dfdbfe4fadaa5e4828dde56b8f30c/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9fe27797e37f4642def3994fea218e7ddb7c94d7f571fff9b06c0f5066ae7aba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9fe27797e37f4642def3994fea218e7ddb7c94d7f571fff9b06c0f5066ae7aba/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a06d3f2bc90ae7f931b0f96b8fca0718b523c0a8e7cb9e6fea5f6a9201479d9a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a06d3f2bc90ae7f931b0f96b8fca0718b523c0a8e7cb9e6fea5f6a9201479d9a/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a56836a111808a0d4adaf9c71b33bc6390b76a15aedd4c96b5c54ec31f364f88/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a56836a111808a0d4adaf9c71b33bc6390b76a15aedd4c96b5c54ec31f364f88/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a655a658b97ac67d1a4601390a3b06ce0901e4bd12c56182b337145e476ad060/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a655a658b97ac67d1a4601390a3b06ce0901e4bd12c56182b337145e476ad060/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a8ab4ef46c9d5296876a382ea4c75b1db6d6d7930fe0ae3587397615ebf918f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a8ab4ef46c9d5296876a382ea4c75b1db6d6d7930fe0ae3587397615ebf918f4/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bbba1dc4118eac14545671fe4575ef65fb40fdd43ef94fba6642f7afcea5810e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bbba1dc4118eac14545671fe4575ef65fb40fdd43ef94fba6642f7afcea5810e/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d61fd4b72b62bcedeb1810130637db8197fa3ad4e426a795f91c9b8f2327d1e1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d61fd4b72b62bcedeb1810130637db8197fa3ad4e426a795f91c9b8f2327d1e1/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d70c75e394efe4b1b2216393091ec3e7cf21149ceae322a6e715b298e713dbac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d70c75e394efe4b1b2216393091ec3e7cf21149ceae322a6e715b298e713dbac/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e42867f37947e2331c4b1764ead6fca0dee5f574a15c6803fa80a1d6fb71f59a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e42867f37947e2331c4b1764ead6fca0dee5f574a15c6803fa80a1d6fb71f59a/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f0cc702f1f07050383c2518b83f462e244a58ce3c3c9a431fdb7f186a57dce6d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f0cc702f1f07050383c2518b83f462e244a58ce3c3c9a431fdb7f186a57dce6d/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fc9e9f0e8e7de5163bf21bc8f42e80dd6f1dc238753973fd2fc6fac5adabb845/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fc9e9f0e8e7de5163bf21bc8f42e80dd6f1dc238753973fd2fc6fac5adabb845/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fdd0cdaef89fda618b32966daf0807210809105824091b7c26732c5089116bf5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fdd0cdaef89fda618b32966daf0807210809105824091b7c26732c5089116bf5/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ffaef20d91077fb6da2ed6f19bd456f0210619e5689b483e6f58bf2166c4f1e4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ffaef20d91077fb6da2ed6f19bd456f0210619e5689b483e6f58bf2166c4f1e4/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05f22644-0884-434d-b56b-193d9554b505/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/05f22644-0884-434d-b56b-193d9554b505/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05f22644-0884-434d-b56b-193d9554b505/volumes/kubernetes.io~projected/kube-api-access-qh4r9:{mountpoint:/var/lib/kubelet/pods/05f22644-0884-434d-b56b-193d9554b505/volumes/kubernetes.io~projected/kube-api-access-qh4r9 major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05f22644-0884-434d-b56b-193d9554b505/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/05f22644-0884-434d-b56b-193d9554b505/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05f22644-0884-434d-b56b-193d9554b505/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/05f22644-0884-434d-b56b-193d9554b505/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08b2f787-ad25-47f9-8ef8-724308458f88/volumes/kubernetes.io~projected/kube-api-access-hv49q:{mountpoint:/var/lib/kubelet/pods/08b2f787-ad25-47f9-8ef8-724308458f88/volumes/kubernetes.io~projected/kube-api-access-hv49q major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/111798c9-6959-4dcb-9a8a-db3037f7d662/volumes/kubernetes.io~projected/kube-api-access-pgvld:{mountpoint:/var/lib/kubelet/pods/111798c9-6959-4dcb-9a8a-db3037f7d662/volumes/kubernetes.io~projected/kube-api-access-pgvld major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/214f4f91-4d6e-46e0-ac1f-d23935cd7215/volumes/kubernetes.io~projected/kube-api-access-nx2vz:{mountpoint:/var/lib/kubelet/pods/214f4f91-4d6e-46e0-ac1f-d23935cd7215/volumes/kubernetes.io~projected/kube-api-access-nx2vz major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/309f81fb-9b40-4053-a060-8672a7c84f77/volumes/kubernetes.io~projected/kube-api-access-tnvlx:{mountpoint:/var/lib/kubelet/pods/309f81fb-9b40-4053-a060-8672a7c84f77/volumes/kubernetes.io~projected/kube-api-access-tnvlx major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/309f81fb-9b40-4053-a060-8672a7c84f77/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/309f81fb-9b40-4053-a060-8672a7c84f77/volumes/kubernetes.io~secret/signing-key major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc/volumes/kubernetes.io~projected/kube-api-access-zgf7l:{mountpoint:/var/lib/kubelet/pods/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc/volumes/kubernetes.io~projected/kube-api-access-zgf7l major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5040db9b-91d3-4d89-adaa-d1394cc3a617/volumes/kubernetes.io~projected/kube-api-access-64rvp:{mountpoint:/var/lib/kubelet/pods/5040db9b-91d3-4d89-adaa-d1394cc3a617/volumes/kubernetes.io~projected/kube-api-access-64rvp major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5040db9b-91d3-4d89-adaa-d1394cc3a617/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/5040db9b-91d3-4d89-adaa-d1394cc3a617/volumes/kubernetes.io~secret/serving-cert major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50f563bf-3336-49c6-9708-041c41b41c53/volumes/kubernetes.io~projected/kube-api-access-ss27z:{mountpoint:/var/lib/kubelet/pods/50f563bf-3336-49c6-9708-041c41b41c53/volumes/kubernetes.io~projected/kube-api-access-ss27z major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/533fb89e-0b69-4c79-9cbf-7baa5f725219/volumes/kubernetes.io~projected/kube-api-access-krgxb:{mountpoint:/var/lib/kubelet/pods/533fb89e-0b69-4c79-9cbf-7baa5f725219/volumes/kubernetes.io~projected/kube-api-access-krgxb major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/533fb89e-0b69-4c79-9cbf-7baa5f725219/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/533fb89e-0b69-4c79-9cbf-7baa5f725219/volumes/kubernetes.io~secret/default-certificate major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/533fb89e-0b69-4c79-9cbf-7baa5f725219/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/533fb89e-0b69-4c79-9cbf-7baa5f725219/volumes/kubernetes.io~secret/stats-auth major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7187a44e-c9e7-4d2d-be9c-f99a11c1401b/volumes/kubernetes.io~projected/kube-api-access-l4vlt:{mountpoint:/var/lib/kubelet/pods/7187a44e-c9e7-4d2d-be9c-f99a11c1401b/volumes/kubernetes.io~projected/kube-api-access-l4vlt major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78c7e317-9d58-4752-9804-92ac45b5afb7/volumes/kubernetes.io~projected/kube-api-access-m8nqb:{mountpoint:/var/lib/kubelet/pods/78c7e317-9d58-4752-9804-92ac45b5afb7/volumes/kubernetes.io~projected/kube-api-access-m8nqb major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80cae79e-b60f-4efe-bb58-e77753a457e5/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/80cae79e-b60f-4efe-bb58-e77753a457e5/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80cae79e-b60f-4efe-bb58-e77753a457e5/volumes/kubernetes.io~projected/kube-api-access-kjn87:{mountpoint:/var/lib/kubelet/pods/80cae79e-b60f-4efe-bb58-e77753a457e5/volumes/kubernetes.io~projected/kube-api-access-kjn87 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80cae79e-b60f-4efe-bb58-e77753a457e5/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/80cae79e-b60f-4efe-bb58-e77753a457e5/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88178646-9467-4225-b362-578f31454922/volumes/kubernetes.io~projected/kube-api-access-2w5vd:{mountpoint:/var/lib/kubelet/pods/88178646-9467-4225-b362-578f31454922/volumes/kubernetes.io~projected/kube-api-access-2w5vd major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf/volumes/kubernetes.io~projected/kube-api-access-tsmc8:{mountpoint:/var/lib/kubelet/pods/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf/volumes/kubernetes.io~projected/kube-api-access-tsmc8 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf/volumes/kubernetes.io~secret/serving-cert major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88d9611e-c7ed-490b-b401-530d1649c873/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/88d9611e-c7ed-490b-b401-530d1649c873/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89ed3d9f-e057-40a3-add4-d6374380be1f/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/89ed3d9f-e057-40a3-add4-d6374380be1f/volumes/kubernetes.io~secret/agent-certs major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b7865f8-cfd2-4938-8621-92b63faee7af/volumes/kubernetes.io~projected/kube-api-access-llcbk:{mountpoint:/var/lib/kubelet/pods/8b7865f8-cfd2-4938-8621-92b63faee7af/volumes/kubernetes.io~projected/kube-api-access-llcbk major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91fe9e06-0c30-4000-bddb-2ba89cd048cc/volumes/kubernetes.io~projected/kube-api-access-94tlh:{mountpoint:/var/lib/kubelet/pods/91fe9e06-0c30-4000-bddb-2ba89cd048cc/volumes/kubernetes.io~projected/kube-api-access-94tlh major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91fe9e06-0c30-4000-bddb-2ba89cd048cc/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/91fe9e06-0c30-4000-bddb-2ba89cd048cc/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f908129-4500-4433-871c-032c29a55dd9/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/9f908129-4500-4433-871c-032c29a55dd9/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f908129-4500-4433-871c-032c29a55dd9/volumes/kubernetes.io~projected/kube-api-access-nqftg:{mountpoint:/var/lib/kubelet/pods/9f908129-4500-4433-871c-032c29a55dd9/volumes/kubernetes.io~projected/kube-api-access-nqftg major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f908129-4500-4433-871c-032c29a55dd9/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/9f908129-4500-4433-871c-032c29a55dd9/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f908129-4500-4433-871c-032c29a55dd9/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/9f908129-4500-4433-871c-032c29a55dd9/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0790a28-d605-4c0c-a652-41ac6ef22e09/volumes/kubernetes.io~projected/kube-api-access-4dx86:{mountpoint:/var/lib/kubelet/pods/a0790a28-d605-4c0c-a652-41ac6ef22e09/volumes/kubernetes.io~projected/kube-api-access-4dx86 major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b87eabe7-2403-4534-987b-ce6ecc6fda94/volumes/kubernetes.io~projected/kube-api-access-g2gcf:{mountpoint:/var/lib/kubelet/pods/b87eabe7-2403-4534-987b-ce6ecc6fda94/volumes/kubernetes.io~projected/kube-api-access-g2gcf major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1e8d4f3-e4a1-416c-bbc7-2c76063bfc24/volumes/kubernetes.io~projected/kube-api-access-7fxk9:{mountpoint:/var/lib/kubelet/pods/c1e8d4f3-e4a1-416c-bbc7-2c76063bfc24/volumes/kubernetes.io~projected/kube-api-access-7fxk9 major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1e8d4f3-e4a1-416c-bbc7-2c76063bfc24/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/c1e8d4f3-e4a1-416c-bbc7-2c76063bfc24/volumes/kubernetes.io~secret/cert major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9732ee5-c72e-44c6-9917-49cc9f81ffe0/volumes/kubernetes.io~projected/kube-api-access-bb792:{mountpoint:/var/lib/kubelet/pods/c9732ee5-c72e-44c6-9917-49cc9f81ffe0/volumes/kubernetes.io~projected/kube-api-access-bb792 major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce6c7b90-6b06-4c81-bf95-539b71e2aaed/volumes/kubernetes.io~projected/kube-api-access-8tpmm:{mountpoint:/var/lib/kubelet/pods/ce6c7b90-6b06-4c81-bf95-539b71e2aaed/volumes/kubernetes.io~projected/kube-api-access-8tpmm major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce6c7b90-6b06-4c81-bf95-539b71e2aaed/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/ce6c7b90-6b06-4c81-bf95-539b71e2aaed/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5312b70-82df-4e75-a300-b8e1d63a75f3/volumes/kubernetes.io~projected/kube-api-access-5wb98:{mountpoint:/var/lib/kubelet/pods/d5312b70-82df-4e75-a300-b8e1d63a75f3/volumes/kubernetes.io~projected/kube-api-access-5wb98 major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d83a8ce1-0cdc-410c-87fb-bfb640dfe702/volumes/kubernetes.io~projected/kube-api-access-mkqqp:{mountpoint:/var/lib/kubelet/pods/d83a8ce1-0cdc-410c-87fb-bfb640dfe702/volumes/kubernetes.io~projected/kube-api-access-mkqqp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9d686e0-6edc-4db0-ab0d-4fd855c2e332/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/d9d686e0-6edc-4db0-ab0d-4fd855c2e332/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9d686e0-6edc-4db0-ab0d-4fd855c2e332/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/d9d686e0-6edc-4db0-ab0d-4fd855c2e332/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9d686e0-6edc-4db0-ab0d-4fd855c2e332/volumes/kubernetes.io~projected/kube-api-access-cwzfx:{mountpoint:/var/lib/kubelet/pods/d9d686e0-6edc-4db0-ab0d-4fd855c2e332/volumes/kubernetes.io~projected/kube-api-access-cwzfx major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea45e6d2-1a47-4956-b652-19b428800f42/volumes/kubernetes.io~projected/kube-api-access-h5zcp:{mountpoint:/var/lib/kubelet/pods/ea45e6d2-1a47-4956-b652-19b428800f42/volumes/kubernetes.io~projected/kube-api-access-h5zcp major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee380d52-b71c-4839-8835-078d98a5e0ee/volumes/kubernetes.io~projected/kube-api-access-9l2sr:{mountpoint:/var/lib/kubelet/pods/ee380d52-b71c-4839-8835-078d98a5e0ee/volumes/kubernetes.io~projected/kube-api-access-9l2sr major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3f7114f-f896-4fc9-b98a-e47574e7720d/volumes/kubernetes.io~projected/kube-api-access-wsx89:{mountpoint:/var/lib/kubelet/pods/f3f7114f-f896-4fc9-b98a-e47574e7720d/volumes/kubernetes.io~projected/kube-api-access-wsx89 major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4913de8-6db5-413a-938f-abc50a5460e7/volumes/kubernetes.io~projected/kube-api-access-d8hdc:{mountpoint:/var/lib/kubelet/pods/f4913de8-6db5-413a-938f-abc50a5460e7/volumes/kubernetes.io~projected/kube-api-access-d8hdc major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc3bbdf6-9dad-4519-ae08-faf3cad565e2/volumes/kubernetes.io~projected/kube-api-access-vzql8:{mountpoint:/var/lib/kubelet/pods/fc3bbdf6-9dad-4519-ae08-faf3cad565e2/volumes/kubernetes.io~projected/kube-api-access-vzql8 major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe578b91-829c-450a-a60a-0f4e4b6e8c63/volumes/kubernetes.io~projected/kube-api-access-m6kxw:{mountpoint:/var/lib/kubelet/pods/fe578b91-829c-450a-a60a-0f4e4b6e8c63/volumes/kubernetes.io~projected/kube-api-access-m6kxw major:0 minor:328 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/04de89569897ab4e33efba7391649bd8e0953b39342c50e084b0d17ce90d9a1e/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/26baa8977b80e58f2797da4b1b158830162da94770f921e98713f4090182ce4c/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/ec9fe987e77a971ad4da62ae565160a3ca3534e9355b647875f8dbd520da9369/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/77c5c896fd31115511f1c8b29ea988f74aa5ece322c07573c163019579607ce2/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/4c60e4d2fb92925a7dcd2667240c1a7e06fcb776dc353e824e368b89f295f0c1/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-118:{mountpoint:/var/lib/containers/storage/overlay/d4f4a1ab8e93db407921d0294bbcfe000a6dbe486be25fdacf3fe3937f0703e8/merged major:0 minor:118 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/f4a954dff2cc0a4b4a742284b5fcf45175360f30d67a27b6be44679fb96006b9/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/4c8007885ed8f79b3056050f7f5f28cc4670c36ef547c53e04c7327fd20a50a3/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/24c2630f18551cf0b7969e929abcd185adccb9c317ea0af4e929ce60b4636a2f/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/5331875a3af4948c5074e0c648b3926c3638607b23de7d93860936b06e9b77f4/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/b2d10c8f7069abb0eccab5dc68cfa6e0e051fedb1e7006865969a6a32af03c40/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/e7def9c5abff269aed608909b00a84130d10d0c14fb06771f45d55486dc5d7fe/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/6ad340bcee75f9ce591be350ab87031d843393889190e21fdfd4c4f317b3aa92/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/77b4c4affb02b3d4290f601e8a537acc977de0006bf46452ec5d7214d232b7e3/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/009b6426b5af01e3188ff62b67225dd6e7a9008fb968572a2cd612ff6002e77b/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/9517c3f5222e7409b0194bcf94e7482f135b5068d2a58e8659ec75217087eeec/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/b93e41e7e7ec1bac6675aa9e5dce2a00740972a7604dcaeb7e71859b45714490/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/04b5b7d641c1fb1dae3181d50f4df3682b5028afe2d205f68cadb6946493d901/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/c601856e27c2092bde1d375bee21dd58019c44beee38a9fb8b1b5782eb9b9cdd/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/283807d8e0a79a30d83d5350c65a2aa1cbff1404cf7271be9cdd626e50988162/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/5a4fb6f299ef1c5c9b6c544fb335cea9442b36de72a25b2723787fb3eca4a3b2/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/a4e93507b629e60ddea27cce774f248661b692320737d0f150988c142b6530a4/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/33fa80a1da93358b21a718c4846dfde14ec350a3766b9d789cbc14bcb0bf4364/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/e180f50dd64b9c97bcbeb68d3174d3355f0fe79f5b0b61dfe44249c91fbeed01/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/82e0e6b68f989961d14f3f5c88c14823172993b9c7208367ecc7d65b809c61fb/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/1ab3b89f3e63294d6299ed19868cc90b0e11d63c28a223a2a173e50b3bdad9ab/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/9531ab10ef0d05487ac871c653202ba596f3fca6dad26d471a0bb503b7bfef59/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/b57baf917a9c6ede699a814e8723e4cf59bdb4f9ac24f4abae9f90c322cbad49/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/fe5e900d826059083d242292783f5b16c589d5178d84a37346bc75a2783d9918/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/1a8902006db365b6c33f165fec10066d8870d830e065de3ac907a7cedbdb853a/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/5745e5c16ddf1963a07868023cee50cc194c821ac2ba878197e8d13cddc4a4a7/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/f4ffa6d1895ff31379f1a83f321b00f6a2dd1d463bcd6b5eb0968ebe196e0c33/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/2406e022ca5e777146e080700fb58bc4fbb4d913af1446a11ba26494eae6a016/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/4ce4234ea61c3de921a33375cbd32c188537a112d56cac2ca3c780560eeedbf7/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/bbc8e177c9cb3ce8b02b7a14001747382ac0d93d59647d5a8cc30b9b8a138866/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/bc3331119a7fe8ea17064cd0a7d2ff79afe76a161241f5b50c78d0fc96ad3509/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/18cfebf56488fbd30effa6840d451e1cc4bf09f2e600d2f9d6a39ea430747d05/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/b2e697ad6f102c70af5fc734689f21721198926cffee0f90062d65d8b4f18264/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/28a32b99d2f4f9f8d7be04ddf5c8545091f25d9468d59612b541e647b12be16b/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/b618daf138bfca10605bde068198c9e3952a23623f3d9bc3ef8f60b3938bc0aa/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/f25a30a40f3c23ca73ac6cbb251653741e7b45c06c70c52f9f380905f7da9033/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/7d87d0851d685e8d5ea1f9daf4ddaf8ef2d22a034f56f09498dbbf7efee0a94d/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/eec4e0a0f8f97464f45ae1c8ad86f41d593e177b9f85740cd0498b2ab08efaf9/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/63c7b56466a39d5cdc1e399dd02ac5a42512c290afa6f6d27301ff06a6744145/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/eed4714124bdd129e61f9e4f7b416b0676cac359769bbcc96726e4445da3169d/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/a7da7ad5bbf87863acf2fcb7cd5718c0ebeb5ce25ab57dce9779aa767efd03fc/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/d066dbd5a247a47522ccf561ddc3e19452cb85a54d66e6c4cfd04c6c8f122387/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/c501f1fe17517bb7ef9240e98b2fde035a4aeeccf059c9c4f6d7ace29edbf836/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/462770add0ebe63b2909176e3b3ce43856ddd3720085126c9cf214bb8e53eb87/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/b7236f6ed70800a87a19d6e8ab66a55c5ba1d554f02e3efb02f7453d00b157ce/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-373:{mountpoint:/var/lib/containers/storage/overlay/4052e89f0a606418150ccf77e9b963a1148e17b3b8824395095c56cec122fadc/merged major:0 minor:373 fsType:overlay blockSize:0} overlay_0-384:{mountpoint:/var/lib/containers/storage/overlay/b31629b381f52b3850d53ec1592398635e33688c947d4835d5a75eedcc4df6d2/merged major:0 minor:384 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/a6a1749dc632d9e65aa03382c60218a98369f106acc84a50265ebdbe728b1f24/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/2ceb65a6682313ebab0e4bdb82a87f37633afda4715af08f7053c50173d7b0fa/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/687fb78fb583abfc0c5311867ba3a880b95db7cfd20139365323ac2dba91df3f/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/75ce2e4e7f32d6e5303e94330237150d2bfc0b701be0bc434f8a0dbd048b77f2/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/c2abddbfc54a2b76a9d15080808e2f7294782596419377e2a90817890c6bc018/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/76fc29a2c97b89f0e37b28636929cef4134e35c8aebf2aa093aa7ba43c2af7a1/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/e866188ad110ba0fd471dd24de187ba2df43428fd5773b618488d16647e828ae/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/8126640b6b2cead91598df71adccda8bfe3e8bd3fa158ccf8f8f30f75e10b7ad/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/8dafd7b8bddc2c6156eb75afecc23c82f38f8776e550abcbbf33cf93ad5db774/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/8d5605f4db3115cc40dc5007e39e93e51f05276272ec17f398d9a570ee9ce248/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 20:26:28.145535 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.145136    5071 manager.go:217] Machine: {Timestamp:2026-10-30 20:26:28.144663948 +0000 UTC m=+0.061685476 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101217 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bd3fa94c1a01fdf528a40e952f581 SystemUUID:ec2bd3fa-94c1-a01f-df52-8a40e952f581 BootID:d2547233-e9df-416f-9af6-3c280c55b54c Filesystems:[{Device:overlay_0-118 DeviceMajor:0 DeviceMinor:118 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/05f22644-0884-434d-b56b-193d9554b505/volumes/kubernetes.io~projected/kube-api-access-qh4r9 DeviceMajor:0 DeviceMinor:228 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c9732ee5-c72e-44c6-9917-49cc9f81ffe0/volumes/kubernetes.io~projected/kube-api-access-bb792 DeviceMajor:0 DeviceMinor:287 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9fe27797e37f4642def3994fea218e7ddb7c94d7f571fff9b06c0f5066ae7aba/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/309f81fb-9b40-4053-a060-8672a7c84f77/volumes/kubernetes.io~projected/kube-api-access-tnvlx DeviceMajor:0 DeviceMinor:361 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf/volumes/kubernetes.io~projected/kube-api-access-tsmc8 DeviceMajor:0 DeviceMinor:226 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a0790a28-d605-4c0c-a652-41ac6ef22e09/volumes/kubernetes.io~projected/kube-api-access-4dx86 DeviceMajor:0 DeviceMinor:229 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/50f563bf-3336-49c6-9708-041c41b41c53/volumes/kubernetes.io~projected/kube-api-access-ss27z DeviceMajor:0 DeviceMinor:336 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3ce759cd9d7ce2099096ca884b37156c6e61e84eae795065d82f7c8b5d1a3d18/userdata/shm DeviceMajor:0 DeviceMinor:338 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c1e8d4f3-e4a1-416c-bbc7-2c76063bfc24/volumes/kubernetes.io~projected/kube-api-access-7fxk9 DeviceMajor:0 DeviceMinor:233 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ea45e6d2-1a47-4956-b652-19b428800f42/volumes/kubernetes.io~projected/kube-api-access-h5zcp DeviceMajor:0 DeviceMinor:51 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/533fb89e-0b69-4c79-9cbf-7baa5f725219/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5040db9b-91d3-4d89-adaa-d1394cc3a617/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:218 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f0cc702f1f07050383c2518b83f462e244a58ce3c3c9a431fdb7f186a57dce6d/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1a20f9bbd988db5a16fa334184144ae38c28e82d8703967417cc8b100e7b2031/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c1e8d4f3-e4a1-416c-bbc7-2c76063bfc24/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:215 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d9d686e0-6edc-4db0-ab0d-4fd855c2e332/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/80cae79e-b60f-4efe-bb58-e77753a457e5/volumes/kubernetes.io~projected/kube-api-access-kjn87 DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0c839da3408e96750313810ca2107e90b21c719237b93eddfac858956709cad8/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/533fb89e-0b69-4c79-9cbf-7baa5f725219/volumes/kubernetes.io~projected/kube-api-access-krgxb DeviceMajor:0 DeviceMinor:205 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9f908129-4500-4433-871c-032c29a55dd9/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:207 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ce6c7b90-6b06-4c81-bf95-539b71e2aaed/volumes/kubernetes.io~projected/kube-api-access-8tpmm DeviceMajor:0 DeviceMinor:209 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/05f22644-0884-434d-b56b-193d9554b505/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:216 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/24bfadd2814fc8daad30a6323ebc6fe65849fec0b2d2c1689868fc5fa2e0ada1/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/50c63a4bd7251e656a4eaea02805ff737213b40bee52f7fe6c2b4e73e32e21a8/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/214f4f91-4d6e-46e0-ac1f-d23935cd7215/volumes/kubernetes.io~projected/kube-api-access-nx2vz DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e42867f37947e2331c4b1764ead6fca0dee5f574a15c6803fa80a1d6fb71f59a/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/08b2f787-ad25-47f9-8ef8-724308458f88/volumes/kubernetes.io~projected/kube-api-access-hv49q DeviceMajor:0 DeviceMinor:206 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7187a44e-c9e7-4d2d-be9c-f99a11c1401b/volumes/kubernetes.io~projected/kube-api-access-l4vlt DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a06d3f2bc90ae7f931b0f96b8fca0718b523c0a8e7cb9e6fea5f6a9201479d9a/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9f908129-4500-4433-871c-032c29a55dd9/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/91fe9e06-0c30-4000-bddb-2ba89cd048cc/volumes/kubernetes.io~projected/kube-api-access-94tlh DeviceMajor:0 DeviceMinor:212 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a655a658b97ac67d1a4601390a3b06ce0901e4bd12c56182b337145e476ad060/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/78c7e317-9d58-4752-9804-92ac45b5afb7/volumes/kubernetes.io~projected/kube-api-access-m8nqb DeviceMajor:0 DeviceMinor:213 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/80cae79e-b60f-4efe-bb58-e77753a457e5/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/b87eabe7-2403-4534-987b-ce6ecc6fda94/volumes/kubernetes.io~projected/kube-api-access-g2gcf DeviceMajor:0 DeviceMinor:334 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9f908129-4500-4433-871c-032c29a55dd9/volumes/kubernetes.io~projected/kube-api-access-nqftg DeviceMajor:0 DeviceMinor:214 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8a1271576ec3b8c9517dd5d184a1f8c2c1b2c19190b24ba6ef7ae3a9cf805ea4/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/89ed3d9f-e057-40a3-add4-d6374380be1f/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:49 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/111798c9-6959-4dcb-9a8a-db3037f7d662/volumes/kubernetes.io~projected/kube-api-access-pgvld DeviceMajor:0 DeviceMinor:54 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/533fb89e-0b69-4c79-9cbf-7baa5f725219/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5040db9b-91d3-4d89-adaa-d1394cc3a617/volumes/kubernetes.io~projected/kube-api-access-64rvp DeviceMajor:0 DeviceMinor:232 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fdd0cdaef89fda618b32966daf0807210809105824091b7c26732c5089116bf5/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/30316f990be28a3c33df9b2c59fd5df78eb099b0430afed2ca8b4801cd011ff7/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d61fd4b72b62bcedeb1810130637db8197fa3ad4e426a795f91c9b8f2327d1e1/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/05f22644-0884-434d-b56b-193d9554b505/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:230 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/38fc45f90c3b66e3c7d6b46a31d34de20e8ba9732b4a3b5841ea69328045fbc5/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9f908129-4500-4433-871c-032c29a55dd9/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/88d9611e-c7ed-490b-b401-530d1649c873/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:282 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fc9e9f0e8e7de5163bf21bc8f42e80dd6f1dc238753973fd2fc6fac5adabb845/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b7865f8-cfd2-4938-8621-92b63faee7af/volumes/kubernetes.io~projected/kube-api-access-llcbk DeviceMajor:0 DeviceMinor:52 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d83a8ce1-0cdc-410c-87fb-bfb640dfe702/volumes/kubernetes.io~projected/kube-api-access-mkqqp DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a8ab4ef46c9d5296876a382ea4c75b1db6d6d7930fe0ae3587397615ebf918f4/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8887ae7a21f88fa23debe43d42535d7b88646ed33ceb9eab7ef9c33c38edc394/userdata/shm DeviceMajor:0 DeviceMinor:333 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fc3bbdf6-9dad-4519-ae08-faf3cad565e2/volumes/kubernetes.io~projected/kube-api-access-vzql8 DeviceMajor:0 DeviceMinor:227 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d70c75e394efe4b1b2216393091ec3e7cf21149ceae322a6e715b298e713dbac/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/309f81fb-9b40-4053-a060-8672a7c84f77/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:356 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/80cae79e-b60f-4efe-bb58-e77753a457e5/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:47 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ffaef20d91077fb6da2ed6f19bd456f0210619e5689b483e6f58bf2166c4f1e4/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ce6c7b90-6b06-4c81-bf95-539b71e2aaed/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f4913de8-6db5-413a-938f-abc50a5460e7/volumes/kubernetes.io~projected/kube-api-access-d8hdc DeviceMajor:0 DeviceMinor:210 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:217 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc/volumes/kubernetes.io~projected/kube-api-access-zgf7l DeviceMajor:0 DeviceMinor:231 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/893a77ea05e781f02d98656c4e2afe76248c31558482172f870102e7960d2962/userdata/shm DeviceMajor:0 DeviceMinor:63 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/05f22644-0884-434d-b56b-193d9554b505/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:219 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f5546fcc9c0952128d607267dd092c06ca09b46cec6046bab3a3c738e3040d4/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f982210565e5738a437f8b5a6bea6e6a28dfdbfe4fadaa5e4828dde56b8f30c/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ee380d52-b71c-4839-8835-078d98a5e0ee/volumes/kubernetes.io~projected/kube-api-access-9l2sr DeviceMajor:0 DeviceMinor:337 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/88178646-9467-4225-b362-578f31454922/volumes/kubernetes.io~projected/kube-api-access-2w5vd DeviceMajor:0 DeviceMinor:53 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/74f6fb01bca34a911a5b4074647d79ef03b18d23b55e713aca7d8b452344a7ae/userdata/shm DeviceMajor:0 DeviceMinor:220 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bbba1dc4118eac14545671fe4575ef65fb40fdd43ef94fba6642f7afcea5810e/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fe578b91-829c-450a-a60a-0f4e4b6e8c63/volumes/kubernetes.io~projected/kube-api-access-m6kxw DeviceMajor:0 DeviceMinor:328 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d9d686e0-6edc-4db0-ab0d-4fd855c2e332/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f3f7114f-f896-4fc9-b98a-e47574e7720d/volumes/kubernetes.io~projected/kube-api-access-wsx89 DeviceMajor:0 DeviceMinor:211 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-384 DeviceMajor:0 DeviceMinor:384 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a56836a111808a0d4adaf9c71b33bc6390b76a15aedd4c96b5c54ec31f364f88/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-373 DeviceMajor:0 DeviceMinor:373 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/91fe9e06-0c30-4000-bddb-2ba89cd048cc/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d5312b70-82df-4e75-a300-b8e1d63a75f3/volumes/kubernetes.io~projected/kube-api-access-5wb98 DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d9d686e0-6edc-4db0-ab0d-4fd855c2e332/volumes/kubernetes.io~projected/kube-api-access-cwzfx DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:24bfadd2814fc8d MacAddress:5e:71:81:4b:d1:e0 Speed:10000 Mtu:8901} {Name:30316f990be28a3 MacAddress:5a:8d:d5:1a:24:3f Speed:10000 Mtu:8901} {Name:38fc45f90c3b66e MacAddress:f6:35:3e:f6:76:73 Speed:10000 Mtu:8901} {Name:3ce759cd9d7ce20 MacAddress:86:61:1c:15:5d:4c Speed:10000 Mtu:8901} {Name:50c63a4bd7251e6 MacAddress:d6:a9:dd:47:2f:8a Speed:10000 Mtu:8901} {Name:74f6fb01bca34a9 MacAddress:4e:c0:cf:d2:8a:03 Speed:10000 Mtu:8901} {Name:8887ae7a21f88fa MacAddress:d2:34:0b:99:6d:48 Speed:10000 Mtu:8901} {Name:8a1271576ec3b8c MacAddress:b2:47:9c:f5:6f:37 Speed:10000 Mtu:8901} {Name:9f982210565e573 MacAddress:9a:7b:71:8c:df:eb Speed:10000 Mtu:8901} {Name:9fe27797e37f464 MacAddress:da:21:ea:ce:1a:cf Speed:10000 Mtu:8901} {Name:a56836a111808a0 MacAddress:62:f7:e4:78:1a:54 Speed:10000 Mtu:8901} {Name:a655a658b97ac67 MacAddress:c6:9d:75:4d:56:cc Speed:10000 Mtu:8901} {Name:bbba1dc4118eac1 MacAddress:32:49:7a:ef:70:18 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:66:29:2f:86:cf Speed:0 Mtu:9001} {Name:br-int MacAddress:8e:46:5c:da:c8:56 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:66:29:2f:86:cf Speed:0 Mtu:9001} {Name:f0cc702f1f07050 MacAddress:ee:81:98:aa:55:8b Speed:10000 Mtu:8901} {Name:fc9e9f0e8e7de51 MacAddress:62:2f:5b:30:25:40 Speed:10000 Mtu:8901} {Name:fdd0cdaef89fda6 MacAddress:32:2d:de:76:30:68 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:52:f4:dd:f9:b9:75 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:e2:60:f4:bd:02:47 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 20:26:28.145535 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.145538    5071 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 20:26:28.145816 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.145583    5071 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 20:26:28.145978 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.145955    5071 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 20:26:28.146107 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.145979    5071 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-149.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 20:26:28.146147 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.146109    5071 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 20:26:28.146147 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.146117    5071 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 20:26:28.146147 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.146130    5071 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 20:26:28.146147 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.146143    5071 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 20:26:28.146252 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.146220    5071 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 20:26:28.146384 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.146374    5071 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 20:26:28.146495 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.146487    5071 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 20:26:28.146530 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.146522    5071 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 20:26:28.146561 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.146538    5071 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 20:26:28.146561 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.146548    5071 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 20:26:28.146561 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.146558    5071 kubelet.go:405] "Adding apiserver pod source"
Oct 30 20:26:28.146661 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.146566    5071 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 20:26:28.147427 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.147412    5071 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 20:26:28.147427 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.147425    5071 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 20:26:28.147940 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.147921    5071 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 20:26:28.150235 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.150221    5071 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 20:26:28.150235 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.150235    5071 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 20:26:28.150338 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.150331    5071 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 20:26:28.150519 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.150508    5071 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 20:26:28.150553 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.150537    5071 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 20:26:28.150645 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.150635    5071 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 20:26:28.150675 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.150651    5071 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 20:26:28.150675 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.150657    5071 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 20:26:28.150675 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.150663    5071 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 20:26:28.150755 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.150680    5071 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 20:26:28.150755 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.150686    5071 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 20:26:28.150755 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.150692    5071 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 20:26:28.150755 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.150698    5071 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 20:26:28.150755 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.150705    5071 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 20:26:28.150755 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.150711    5071 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 20:26:28.150755 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.150726    5071 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 20:26:28.150755 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.150735    5071 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 20:26:28.150982 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.150771    5071 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 20:26:28.150982 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.150777    5071 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 20:26:28.154237 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.154225    5071 server.go:1263] "Started kubelet"
Oct 30 20:26:28.154329 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.154301    5071 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 20:26:28.154524 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.154479    5071 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 20:26:28.155211 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.154547    5071 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 20:26:28.155211 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.155069    5071 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 20:26:28.154760 ip-10-0-139-149 systemd[1]: Started Kubernetes Kubelet.
Oct 30 20:26:28.155629 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.155586    5071 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 20:26:28.161851 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.159592    5071 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 20:26:28.161851 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.159607    5071 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 20:26:28.161851 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.159679    5071 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:20:47 +0000 UTC" deadline="2028-06-05 18:12:00.937186829 +0000 UTC"
Oct 30 20:26:28.161851 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.159698    5071 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14013h45m32.777491852s"
Oct 30 20:26:28.161851 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.159852    5071 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 20:26:28.161851 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.159866    5071 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 20:26:28.161851 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.159972    5071 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 20:26:28.161851 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:28.160114    5071 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-149.us-west-1.compute.internal\" not found"
Oct 30 20:26:28.162319 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.162302    5071 factory.go:55] Registering systemd factory
Oct 30 20:26:28.162369 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.162322    5071 factory.go:223] Registration of the systemd container factory successfully
Oct 30 20:26:28.162722 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.162699    5071 factory.go:153] Registering CRI-O factory
Oct 30 20:26:28.162722 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.162722    5071 factory.go:223] Registration of the crio container factory successfully
Oct 30 20:26:28.162904 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.162776    5071 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 20:26:28.162904 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.162797    5071 factory.go:103] Registering Raw factory
Oct 30 20:26:28.162904 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.162811    5071 manager.go:1199] Started watching for new ooms in manager
Oct 30 20:26:28.163069 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.162938    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05f22644-0884-434d-b56b-193d9554b505" volumeName="kubernetes.io/secret/05f22644-0884-434d-b56b-193d9554b505-installation-pull-secrets" seLinuxMountContext=""
Oct 30 20:26:28.163069 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.162973    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="111798c9-6959-4dcb-9a8a-db3037f7d662" volumeName="kubernetes.io/configmap/111798c9-6959-4dcb-9a8a-db3037f7d662-iptables-alerter-script" seLinuxMountContext=""
Oct 30 20:26:28.163069 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.162982    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c7e75b7-c3cd-4791-9d39-9120f6fd95dc" volumeName="kubernetes.io/configmap/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-config-volume" seLinuxMountContext=""
Oct 30 20:26:28.163069 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.162991    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80cae79e-b60f-4efe-bb58-e77753a457e5" volumeName="kubernetes.io/configmap/80cae79e-b60f-4efe-bb58-e77753a457e5-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 20:26:28.163069 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.162999    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88178646-9467-4225-b362-578f31454922" volumeName="kubernetes.io/projected/88178646-9467-4225-b362-578f31454922-kube-api-access-2w5vd" seLinuxMountContext=""
Oct 30 20:26:28.163069 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163008    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88d9611e-c7ed-490b-b401-530d1649c873" volumeName="kubernetes.io/secret/88d9611e-c7ed-490b-b401-530d1649c873-original-pull-secret" seLinuxMountContext=""
Oct 30 20:26:28.163069 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163015    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9732ee5-c72e-44c6-9917-49cc9f81ffe0" volumeName="kubernetes.io/projected/c9732ee5-c72e-44c6-9917-49cc9f81ffe0-kube-api-access-bb792" seLinuxMountContext=""
Oct 30 20:26:28.163069 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163022    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce6c7b90-6b06-4c81-bf95-539b71e2aaed" volumeName="kubernetes.io/projected/ce6c7b90-6b06-4c81-bf95-539b71e2aaed-kube-api-access-8tpmm" seLinuxMountContext=""
Oct 30 20:26:28.163069 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163031    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="533fb89e-0b69-4c79-9cbf-7baa5f725219" volumeName="kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-stats-auth" seLinuxMountContext=""
Oct 30 20:26:28.163069 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163038    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8840f0ca-269e-4924-8b3d-5fe6ff4e90cf" volumeName="kubernetes.io/configmap/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 20:26:28.163069 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163045    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f908129-4500-4433-871c-032c29a55dd9" volumeName="kubernetes.io/secret/9f908129-4500-4433-871c-032c29a55dd9-installation-pull-secrets" seLinuxMountContext=""
Oct 30 20:26:28.163069 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163052    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5312b70-82df-4e75-a300-b8e1d63a75f3" volumeName="kubernetes.io/configmap/d5312b70-82df-4e75-a300-b8e1d63a75f3-telemetry-config" seLinuxMountContext=""
Oct 30 20:26:28.163069 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163060    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d83a8ce1-0cdc-410c-87fb-bfb640dfe702" volumeName="kubernetes.io/projected/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-kube-api-access-mkqqp" seLinuxMountContext=""
Oct 30 20:26:28.163069 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163067    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea45e6d2-1a47-4956-b652-19b428800f42" volumeName="kubernetes.io/projected/ea45e6d2-1a47-4956-b652-19b428800f42-kube-api-access-h5zcp" seLinuxMountContext=""
Oct 30 20:26:28.163069 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163076    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee380d52-b71c-4839-8835-078d98a5e0ee" volumeName="kubernetes.io/projected/ee380d52-b71c-4839-8835-078d98a5e0ee-kube-api-access-9l2sr" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163084    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05f22644-0884-434d-b56b-193d9554b505" volumeName="kubernetes.io/secret/05f22644-0884-434d-b56b-193d9554b505-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163092    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c7e75b7-c3cd-4791-9d39-9120f6fd95dc" volumeName="kubernetes.io/empty-dir/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-tmp-dir" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163099    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="533fb89e-0b69-4c79-9cbf-7baa5f725219" volumeName="kubernetes.io/projected/533fb89e-0b69-4c79-9cbf-7baa5f725219-kube-api-access-krgxb" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163106    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4913de8-6db5-413a-938f-abc50a5460e7" volumeName="kubernetes.io/projected/f4913de8-6db5-413a-938f-abc50a5460e7-kube-api-access-d8hdc" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163112    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91fe9e06-0c30-4000-bddb-2ba89cd048cc" volumeName="kubernetes.io/secret/91fe9e06-0c30-4000-bddb-2ba89cd048cc-serving-cert" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163119    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5040db9b-91d3-4d89-adaa-d1394cc3a617" volumeName="kubernetes.io/secret/5040db9b-91d3-4d89-adaa-d1394cc3a617-serving-cert" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163125    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70fa9600-c497-4deb-a133-3cb24f1c8050" volumeName="kubernetes.io/configmap/70fa9600-c497-4deb-a133-3cb24f1c8050-nginx-conf" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163132    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89ed3d9f-e057-40a3-add4-d6374380be1f" volumeName="kubernetes.io/secret/89ed3d9f-e057-40a3-add4-d6374380be1f-agent-certs" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163139    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f908129-4500-4433-871c-032c29a55dd9" volumeName="kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-kube-api-access-nqftg" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163155    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b87eabe7-2403-4534-987b-ce6ecc6fda94" volumeName="kubernetes.io/empty-dir/b87eabe7-2403-4534-987b-ce6ecc6fda94-data-volume" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163164    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce6c7b90-6b06-4c81-bf95-539b71e2aaed" volumeName="kubernetes.io/configmap/ce6c7b90-6b06-4c81-bf95-539b71e2aaed-config" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163172    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8840f0ca-269e-4924-8b3d-5fe6ff4e90cf" volumeName="kubernetes.io/projected/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf-kube-api-access-tsmc8" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163180    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="309f81fb-9b40-4053-a060-8672a7c84f77" volumeName="kubernetes.io/secret/309f81fb-9b40-4053-a060-8672a7c84f77-signing-key" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163182    5071 manager.go:319] Starting recovery of all containers
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163191    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05f22644-0884-434d-b56b-193d9554b505" volumeName="kubernetes.io/configmap/05f22644-0884-434d-b56b-193d9554b505-registry-certificates" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163207    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08b2f787-ad25-47f9-8ef8-724308458f88" volumeName="kubernetes.io/projected/08b2f787-ad25-47f9-8ef8-724308458f88-kube-api-access-hv49q" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163216    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="111798c9-6959-4dcb-9a8a-db3037f7d662" volumeName="kubernetes.io/projected/111798c9-6959-4dcb-9a8a-db3037f7d662-kube-api-access-pgvld" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163223    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f908129-4500-4433-871c-032c29a55dd9" volumeName="kubernetes.io/secret/9f908129-4500-4433-871c-032c29a55dd9-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163231    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9d686e0-6edc-4db0-ab0d-4fd855c2e332" volumeName="kubernetes.io/projected/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-kube-api-access-cwzfx" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163239    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05f22644-0884-434d-b56b-193d9554b505" volumeName="kubernetes.io/empty-dir/05f22644-0884-434d-b56b-193d9554b505-ca-trust-extracted" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163247    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="214f4f91-4d6e-46e0-ac1f-d23935cd7215" volumeName="kubernetes.io/projected/214f4f91-4d6e-46e0-ac1f-d23935cd7215-kube-api-access-nx2vz" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163254    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91fe9e06-0c30-4000-bddb-2ba89cd048cc" volumeName="kubernetes.io/projected/91fe9e06-0c30-4000-bddb-2ba89cd048cc-kube-api-access-94tlh" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163261    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee380d52-b71c-4839-8835-078d98a5e0ee" volumeName="kubernetes.io/empty-dir/ee380d52-b71c-4839-8835-078d98a5e0ee-archives-path" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163268    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05f22644-0884-434d-b56b-193d9554b505" volumeName="kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-kube-api-access-qh4r9" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163276    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="533fb89e-0b69-4c79-9cbf-7baa5f725219" volumeName="kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-default-certificate" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163283    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8840f0ca-269e-4924-8b3d-5fe6ff4e90cf" volumeName="kubernetes.io/empty-dir/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf-snapshots" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163294    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8840f0ca-269e-4924-8b3d-5fe6ff4e90cf" volumeName="kubernetes.io/empty-dir/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf-tmp" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163302    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b7865f8-cfd2-4938-8621-92b63faee7af" volumeName="kubernetes.io/configmap/8b7865f8-cfd2-4938-8621-92b63faee7af-serviceca" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163310    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f908129-4500-4433-871c-032c29a55dd9" volumeName="kubernetes.io/configmap/9f908129-4500-4433-871c-032c29a55dd9-registry-certificates" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163318    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f908129-4500-4433-871c-032c29a55dd9" volumeName="kubernetes.io/configmap/9f908129-4500-4433-871c-032c29a55dd9-trusted-ca" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163325    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1e8d4f3-e4a1-416c-bbc7-2c76063bfc24" volumeName="kubernetes.io/projected/c1e8d4f3-e4a1-416c-bbc7-2c76063bfc24-kube-api-access-7fxk9" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163331    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7187a44e-c9e7-4d2d-be9c-f99a11c1401b" volumeName="kubernetes.io/configmap/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-cni-binary-copy" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163354    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d83a8ce1-0cdc-410c-87fb-bfb640dfe702" volumeName="kubernetes.io/configmap/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-cni-binary-copy" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163366    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9d686e0-6edc-4db0-ab0d-4fd855c2e332" volumeName="kubernetes.io/empty-dir/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-tmp" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163377    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee380d52-b71c-4839-8835-078d98a5e0ee" volumeName="kubernetes.io/configmap/ee380d52-b71c-4839-8835-078d98a5e0ee-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163388    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc3bbdf6-9dad-4519-ae08-faf3cad565e2" volumeName="kubernetes.io/projected/fc3bbdf6-9dad-4519-ae08-faf3cad565e2-kube-api-access-vzql8" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163399    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88178646-9467-4225-b362-578f31454922" volumeName="kubernetes.io/empty-dir/88178646-9467-4225-b362-578f31454922-tmp-dir" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163409    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8840f0ca-269e-4924-8b3d-5fe6ff4e90cf" volumeName="kubernetes.io/configmap/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf-service-ca-bundle" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163420    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b7865f8-cfd2-4938-8621-92b63faee7af" volumeName="kubernetes.io/projected/8b7865f8-cfd2-4938-8621-92b63faee7af-kube-api-access-llcbk" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163431    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3f7114f-f896-4fc9-b98a-e47574e7720d" volumeName="kubernetes.io/configmap/f3f7114f-f896-4fc9-b98a-e47574e7720d-samples-operator-config" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163442    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5040db9b-91d3-4d89-adaa-d1394cc3a617" volumeName="kubernetes.io/projected/5040db9b-91d3-4d89-adaa-d1394cc3a617-kube-api-access-64rvp" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163454    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7187a44e-c9e7-4d2d-be9c-f99a11c1401b" volumeName="kubernetes.io/projected/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-kube-api-access-l4vlt" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163472    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80cae79e-b60f-4efe-bb58-e77753a457e5" volumeName="kubernetes.io/configmap/80cae79e-b60f-4efe-bb58-e77753a457e5-env-overrides" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163483    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3f7114f-f896-4fc9-b98a-e47574e7720d" volumeName="kubernetes.io/projected/f3f7114f-f896-4fc9-b98a-e47574e7720d-kube-api-access-wsx89" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163496    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7187a44e-c9e7-4d2d-be9c-f99a11c1401b" volumeName="kubernetes.io/configmap/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163509    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89ed3d9f-e057-40a3-add4-d6374380be1f" volumeName="kubernetes.io/configmap/89ed3d9f-e057-40a3-add4-d6374380be1f-konnectivity-ca" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163520    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce6c7b90-6b06-4c81-bf95-539b71e2aaed" volumeName="kubernetes.io/secret/ce6c7b90-6b06-4c81-bf95-539b71e2aaed-serving-cert" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163531    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d83a8ce1-0cdc-410c-87fb-bfb640dfe702" volumeName="kubernetes.io/configmap/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-multus-daemon-config" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163543    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee380d52-b71c-4839-8835-078d98a5e0ee" volumeName="kubernetes.io/configmap/ee380d52-b71c-4839-8835-078d98a5e0ee-service-ca-bundle" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163554    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f908129-4500-4433-871c-032c29a55dd9" volumeName="kubernetes.io/empty-dir/9f908129-4500-4433-871c-032c29a55dd9-ca-trust-extracted" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163566    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="309f81fb-9b40-4053-a060-8672a7c84f77" volumeName="kubernetes.io/configmap/309f81fb-9b40-4053-a060-8672a7c84f77-signing-cabundle" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163577    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5040db9b-91d3-4d89-adaa-d1394cc3a617" volumeName="kubernetes.io/configmap/5040db9b-91d3-4d89-adaa-d1394cc3a617-config" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163589    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50f563bf-3336-49c6-9708-041c41b41c53" volumeName="kubernetes.io/projected/50f563bf-3336-49c6-9708-041c41b41c53-kube-api-access-ss27z" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163600    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78c7e317-9d58-4752-9804-92ac45b5afb7" volumeName="kubernetes.io/projected/78c7e317-9d58-4752-9804-92ac45b5afb7-kube-api-access-m8nqb" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163610    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91fe9e06-0c30-4000-bddb-2ba89cd048cc" volumeName="kubernetes.io/configmap/91fe9e06-0c30-4000-bddb-2ba89cd048cc-trusted-ca" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163622    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8840f0ca-269e-4924-8b3d-5fe6ff4e90cf" volumeName="kubernetes.io/secret/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf-serving-cert" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163634    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7187a44e-c9e7-4d2d-be9c-f99a11c1401b" volumeName="kubernetes.io/configmap/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163645    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05f22644-0884-434d-b56b-193d9554b505" volumeName="kubernetes.io/configmap/05f22644-0884-434d-b56b-193d9554b505-trusted-ca" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163660    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80cae79e-b60f-4efe-bb58-e77753a457e5" volumeName="kubernetes.io/configmap/80cae79e-b60f-4efe-bb58-e77753a457e5-ovnkube-config" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163671    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80cae79e-b60f-4efe-bb58-e77753a457e5" volumeName="kubernetes.io/secret/80cae79e-b60f-4efe-bb58-e77753a457e5-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163683    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0790a28-d605-4c0c-a652-41ac6ef22e09" volumeName="kubernetes.io/projected/a0790a28-d605-4c0c-a652-41ac6ef22e09-kube-api-access-4dx86" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163694    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05f22644-0884-434d-b56b-193d9554b505" volumeName="kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-bound-sa-token" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163708    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c7e75b7-c3cd-4791-9d39-9120f6fd95dc" volumeName="kubernetes.io/projected/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-kube-api-access-zgf7l" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163727    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80cae79e-b60f-4efe-bb58-e77753a457e5" volumeName="kubernetes.io/projected/80cae79e-b60f-4efe-bb58-e77753a457e5-kube-api-access-kjn87" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163746    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe578b91-829c-450a-a60a-0f4e4b6e8c63" volumeName="kubernetes.io/projected/fe578b91-829c-450a-a60a-0f4e4b6e8c63-kube-api-access-m6kxw" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163763    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91fe9e06-0c30-4000-bddb-2ba89cd048cc" volumeName="kubernetes.io/configmap/91fe9e06-0c30-4000-bddb-2ba89cd048cc-config" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163775    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1e8d4f3-e4a1-416c-bbc7-2c76063bfc24" volumeName="kubernetes.io/secret/c1e8d4f3-e4a1-416c-bbc7-2c76063bfc24-cert" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163786    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5312b70-82df-4e75-a300-b8e1d63a75f3" volumeName="kubernetes.io/projected/d5312b70-82df-4e75-a300-b8e1d63a75f3-kube-api-access-5wb98" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163798    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="309f81fb-9b40-4053-a060-8672a7c84f77" volumeName="kubernetes.io/projected/309f81fb-9b40-4053-a060-8672a7c84f77-kube-api-access-tnvlx" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163811    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f908129-4500-4433-871c-032c29a55dd9" volumeName="kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-bound-sa-token" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163840    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b87eabe7-2403-4534-987b-ce6ecc6fda94" volumeName="kubernetes.io/projected/b87eabe7-2403-4534-987b-ce6ecc6fda94-kube-api-access-g2gcf" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163852    5071 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9d686e0-6edc-4db0-ab0d-4fd855c2e332" volumeName="kubernetes.io/empty-dir/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-tuned" seLinuxMountContext=""
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163864    5071 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 20:26:28.163803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.163872    5071 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 20:26:28.185493 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.185463    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8qvb_91fe9e06-0c30-4000-bddb-2ba89cd048cc/console-operator/1.log"
Oct 30 20:26:28.188670 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.188637    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8qvb_91fe9e06-0c30-4000-bddb-2ba89cd048cc/console-operator/1.log"
Oct 30 20:26:28.191708 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.191679    5071 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 20:26:28.193387 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.193363    5071 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 20:26:28.193387 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.193388    5071 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 20:26:28.193503 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.193411    5071 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 20:26:28.193503 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:28.193453    5071 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 20:26:28.196872 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.196853    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q4f4h_80cae79e-b60f-4efe-bb58-e77753a457e5/ovn-acl-logging/0.log"
Oct 30 20:26:28.201267 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.201245    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q4f4h_80cae79e-b60f-4efe-bb58-e77753a457e5/ovn-acl-logging/0.log"
Oct 30 20:26:28.204603 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.204573    5071 generic.go:356] "Generic (PLEG): container finished" podID="c591170baa3135b88257a110e786e9de" containerID="4874c533c2cb70d6e641c0a2aed6db15c4dfa830b376680780873e212fcd2dcf" exitCode=0
Oct 30 20:26:28.213985 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.213963    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8qvb_91fe9e06-0c30-4000-bddb-2ba89cd048cc/console-operator/1.log"
Oct 30 20:26:28.214068 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.213995    5071 generic.go:356] "Generic (PLEG): container finished" podID="91fe9e06-0c30-4000-bddb-2ba89cd048cc" containerID="b21ea8949c9de8f71605d238975ce27033db9c0ec4dff0ed0d0894a5dbcff357" exitCode=255
Oct 30 20:26:28.220695 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.220673    5071 generic.go:356] "Generic (PLEG): container finished" podID="7187a44e-c9e7-4d2d-be9c-f99a11c1401b" containerID="019e1ce8429641b2ed5debe382e0949988fa1698c64bfc9f75a32d1b2a6cfbbc" exitCode=0
Oct 30 20:26:28.220695 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.220695    5071 generic.go:356] "Generic (PLEG): container finished" podID="7187a44e-c9e7-4d2d-be9c-f99a11c1401b" containerID="41ae2317108bb4d64c40d11f6c0137844f6cd4b4ac18dbdf44d473a8bf1c9eb7" exitCode=0
Oct 30 20:26:28.220870 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.220704    5071 generic.go:356] "Generic (PLEG): container finished" podID="7187a44e-c9e7-4d2d-be9c-f99a11c1401b" containerID="a90145f99140af5ab86983c37e5bc80ca15cb9c0bb85b2ad56156f8bc89108e9" exitCode=0
Oct 30 20:26:28.220870 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.220714    5071 generic.go:356] "Generic (PLEG): container finished" podID="7187a44e-c9e7-4d2d-be9c-f99a11c1401b" containerID="cc667eeb6d869e37e03a0b811e13b6c06507e125b5710a0e315d292a12f111c0" exitCode=0
Oct 30 20:26:28.220870 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.220719    5071 generic.go:356] "Generic (PLEG): container finished" podID="7187a44e-c9e7-4d2d-be9c-f99a11c1401b" containerID="909803419acdf3ca60b7a0e44e4095ddacfda4d2ef52f1f447c29bd748d65343" exitCode=0
Oct 30 20:26:28.220870 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.220724    5071 generic.go:356] "Generic (PLEG): container finished" podID="7187a44e-c9e7-4d2d-be9c-f99a11c1401b" containerID="bc09d9b2b8c85f0029a93b2f9f90135349107527cb8a7c6ea6ca16547307926e" exitCode=0
Oct 30 20:26:28.229033 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.229014    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q4f4h_80cae79e-b60f-4efe-bb58-e77753a457e5/ovn-acl-logging/0.log"
Oct 30 20:26:28.229370 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.229347    5071 generic.go:356] "Generic (PLEG): container finished" podID="80cae79e-b60f-4efe-bb58-e77753a457e5" containerID="cb54274f3a3593acf07f66429cd4d519074103fea0a4321ba5607d5f526d82ea" exitCode=1
Oct 30 20:26:28.260188 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:28.260162    5071 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-149.us-west-1.compute.internal\" not found"
Oct 30 20:26:28.266255 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.266233    5071 manager.go:324] Recovery completed
Oct 30 20:26:28.284213 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.284149    5071 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:26:28.285578 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.285556    5071 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-149.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:26:28.285681 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.285648    5071 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-149.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:26:28.285681 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.285668    5071 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-149.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:26:28.288342 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.288323    5071 cpu_manager.go:226] "Starting" policy="none"
Oct 30 20:26:28.288342 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.288339    5071 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:26:28.288449 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.288339    5071 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 20:26:28.288449 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.288440    5071 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 20:26:28.288590 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.288576    5071 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 20:26:28.288634 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.288591    5071 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 20:26:28.288634 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.288607    5071 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 20:26:28.288634 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.288612    5071 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 20:26:28.288634 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.288618    5071 policy_none.go:50] "Start"
Oct 30 20:26:28.288634 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.288627    5071 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 20:26:28.288775 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.288636    5071 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 20:26:28.288775 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.288760    5071 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 20:26:28.288775 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.288766    5071 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 20:26:28.288775 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.288772    5071 policy_none.go:44] "Start"
Oct 30 20:26:28.291884 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.291865    5071 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-149.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:26:28.291985 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.291896    5071 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-149.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:26:28.291985 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.291907    5071 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-149.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:26:28.293561 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:28.293545    5071 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 20:26:28.294945 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.294918    5071 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 20:26:28.297453 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.297440    5071 manager.go:341] "Starting Device Plugin manager"
Oct 30 20:26:28.297511 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:28.297473    5071 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 20:26:28.297511 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.297485    5071 server.go:87] "Starting device plugin registration server"
Oct 30 20:26:28.297742 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.297731    5071 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 20:26:28.297777 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.297742    5071 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 20:26:28.297864 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.297841    5071 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 20:26:28.297982 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.297947    5071 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 20:26:28.298025 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.297995    5071 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 20:26:28.298025 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.298002    5071 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 20:26:28.298136 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.298056    5071 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T20:26:28.297982574Z","UUID":"234e9a08-5cac-4559-82c5-197fd163ae32","Handler":null,"Name":"","Endpoint":""}
Oct 30 20:26:28.299477 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.299462    5071 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 20:26:28.299569 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.299482    5071 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 20:26:28.300481 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:28.300454    5071 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 20:26:28.300597 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:28.300493    5071 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-149.us-west-1.compute.internal\" not found"
Oct 30 20:26:28.353408 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.353380    5071 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:26:28.353408 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.353385    5071 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:26:28.353612 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.353477    5071 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 20:26:28.354799 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.354773    5071 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:26:28.398303 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.398267    5071 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:26:28.399542 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.399526    5071 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-149.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:26:28.399626 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.399555    5071 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-149.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:26:28.399626 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.399567    5071 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-149.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:26:28.399695 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.399634    5071 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:26:28.404704 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.404676    5071 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:26:28.494382 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.494323    5071 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-149.us-west-1.compute.internal"]
Oct 30 20:26:28.495079 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.495037    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-149.us-west-1.compute.internal" event={"ID":"9d9f66ca5c88277c8c76e3090f63988d","Type":"ContainerStarted","Data":"a8ab4ef46c9d5296876a382ea4c75b1db6d6d7930fe0ae3587397615ebf918f4"}
Oct 30 20:26:28.495129 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.495083    5071 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:26:28.495129 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.495091    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal" event={"ID":"c591170baa3135b88257a110e786e9de","Type":"ContainerStarted","Data":"14f4850e366661720e26c84d3938106769611c518258721e1a994af71d14fa48"}
Oct 30 20:26:28.495129 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.495117    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal" event={"ID":"c591170baa3135b88257a110e786e9de","Type":"ContainerDied","Data":"4874c533c2cb70d6e641c0a2aed6db15c4dfa830b376680780873e212fcd2dcf"}
Oct 30 20:26:28.495269 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.495137    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal" event={"ID":"c591170baa3135b88257a110e786e9de","Type":"ContainerStarted","Data":"d61fd4b72b62bcedeb1810130637db8197fa3ad4e426a795f91c9b8f2327d1e1"}
Oct 30 20:26:28.495317 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.495298    5071 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:26:28.516191 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.516165    5071 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:26:28.516297 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.516226    5071 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:26:28.516811 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.516793    5071 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-149.us-west-1.compute.internal" event="NodeNotReady"
Oct 30 20:26:28.516942 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.516844    5071 setters.go:546] "Node became not ready" node="ip-10-0-139-149.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-30T20:26:28Z","lastTransitionTime":"2026-10-30T20:26:28Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Oct 30 20:26:28.523420 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.523401    5071 nodeinfomanager.go:471] existing CSINode "ip-10-0-139-149.us-west-1.compute.internal" is owned by different node (oldNodeID="9b915f5d-0c32-49db-a3f2-d43f8d79bd17", newNodeID=""), cleaning up...
Oct 30 20:26:28.559483 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.559459    5071 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:26:28.559566 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:28.559515    5071 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-149.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:26:28.559655 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.559640    5071 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:26:28.559687 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:28.559675    5071 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:26:28.655889 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.655860    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c591170baa3135b88257a110e786e9de-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal\" (UID: \"c591170baa3135b88257a110e786e9de\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:26:28.655889 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.655891    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c591170baa3135b88257a110e786e9de-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal\" (UID: \"c591170baa3135b88257a110e786e9de\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:26:28.656091 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.655910    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9d9f66ca5c88277c8c76e3090f63988d-config\") pod \"kube-apiserver-proxy-ip-10-0-139-149.us-west-1.compute.internal\" (UID: \"9d9f66ca5c88277c8c76e3090f63988d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:26:28.756017 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.755984    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c591170baa3135b88257a110e786e9de-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal\" (UID: \"c591170baa3135b88257a110e786e9de\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:26:28.756017 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.756014    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c591170baa3135b88257a110e786e9de-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal\" (UID: \"c591170baa3135b88257a110e786e9de\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:26:28.756230 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.756032    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9d9f66ca5c88277c8c76e3090f63988d-config\") pod \"kube-apiserver-proxy-ip-10-0-139-149.us-west-1.compute.internal\" (UID: \"9d9f66ca5c88277c8c76e3090f63988d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:26:28.756230 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.756074    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9d9f66ca5c88277c8c76e3090f63988d-config\") pod \"kube-apiserver-proxy-ip-10-0-139-149.us-west-1.compute.internal\" (UID: \"9d9f66ca5c88277c8c76e3090f63988d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:26:28.756230 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.756102    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c591170baa3135b88257a110e786e9de-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal\" (UID: \"c591170baa3135b88257a110e786e9de\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:26:28.756230 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.756102    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c591170baa3135b88257a110e786e9de-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal\" (UID: \"c591170baa3135b88257a110e786e9de\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:26:28.789060 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.789036    5071 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-149.us-west-1.compute.internal" event="NodeReady"
Oct 30 20:26:28.789168 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:28.789127    5071 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 20:26:29.147504 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.147473    5071 apiserver.go:52] "Watching apiserver"
Oct 30 20:26:29.258194 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.256539    5071 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:26:29.258194 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.257079    5071 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:26:29.369873 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.369783    5071 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:26:29.369873 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.369783    5071 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:26:29.370036 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:29.369914    5071 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-149.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:26:29.370036 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:29.369859    5071 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal"
Oct 30 20:26:29.370164 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.370133    5071 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 20:26:29.370923 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.370892    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-brf75","openshift-console-operator/console-operator-7cd56b8dbc-l8qvb","openshift-kube-storage-version-migrator/migrator-66df5d496b-lfrbc","openshift-ovn-kubernetes/ovnkube-node-q4f4h","kube-system/kube-apiserver-proxy-ip-10-0-139-149.us-west-1.compute.internal","openshift-service-ca/service-ca-5fffc97bdf-6b75b","openshift-service-ca-operator/service-ca-operator-544d749c8c-tp7sc","openshift-image-registry/image-registry-7db8fb6d4-8n6g7","openshift-multus/network-metrics-daemon-zcslg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.us-west-1.compute.internal","openshift-monitoring/cluster-monitoring-operator-884576c57-9rs22","openshift-network-operator/iptables-alerter-kwm7g","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95","openshift-multus/multus-additional-cni-plugins-qxwlb","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-m9n59","openshift-ingress/router-default-666dd86ff-dnnq5","openshift-insights/insights-operator-7c4986d449-mql5m","openshift-dns/node-resolver-c8422","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf","openshift-network-diagnostics/network-check-source-6f78d59f99-bhrtb","openshift-network-console/networking-console-plugin-86c478bb88-4v8mv","openshift-network-diagnostics/network-check-target-pfvnn","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z9tdd","kube-system/global-pull-secret-syncer-bqh9l","openshift-insights/insights-runtime-extractor-tk6nb","openshift-multus/multus-2b77p","openshift-dns/dns-default-h2vh7","openshift-kube-storage-version-migrator/migrator-66df5d496b-hbm7v","openshift-image-registry/node-ca-rdqrc","openshift-insights/periodic-gathering-zmknv-22np8","kube-system/kas-connection-checker-685895c7c9-kgsjn","kube-system/kas-connection-checker-685895c7c9-c5b2k","kube-system/konnectivity-agent-4khkm","openshift-cluster-node-tuning-operator/tuned-59qcr","openshift-ingress-canary/ingress-canary-mrnvn","openshift-image-registry/image-registry-968d68cdf-vwtgc"]
Oct 30 20:26:29.372765 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.372745    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:29.373163 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.373144    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9rs22"
Oct 30 20:26:29.373473 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.373455    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:29.373885 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.373868    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-m9n59"
Oct 30 20:26:29.375925 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.375908    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-4v8mv"
Oct 30 20:26:29.380463 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.380442    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:26:29.385220 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.385201    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:29.385814 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.385800    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h2vh7"
Oct 30 20:26:29.388433 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.388414    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:29.425754 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.425717    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-42gkn\""
Oct 30 20:26:29.425951 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.425935    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 20:26:29.426895 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.426880    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 20:26:29.426895 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.426889    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 20:26:29.428179 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.428166    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 20:26:29.433990 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.433973    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 20:26:29.434076 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.434031    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 20:26:29.434322 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.434303    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 20:26:29.434384 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.434324    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 20:26:29.434418 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.434383    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 20:26:29.434570 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.434549    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:26:29.434570 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.434563    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 20:26:29.434729 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.434574    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 20:26:29.434729 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.434606    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-6lsj6\""
Oct 30 20:26:29.434729 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.434656    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 20:26:29.434729 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.434666    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 20:26:29.434729 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.434679    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-zhmkz\""
Oct 30 20:26:29.434932 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.434749    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 20:26:29.434932 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.434870    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nrwxn\""
Oct 30 20:26:29.434932 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.434875    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 20:26:29.434932 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.434882    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 20:26:29.435060 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.435000    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 20:26:29.435060 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.435024    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 20:26:29.435320 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.435306    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 20:26:29.435390 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.435377    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 20:26:29.435430 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.435383    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 20:26:29.435430 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.435413    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 20:26:29.435604 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.435590    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-d75rn\""
Oct 30 20:26:29.435900 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.435877    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 20:26:29.435954 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.435897    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-lbdrb\""
Oct 30 20:26:29.435954 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.435882    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-dm7dq\""
Oct 30 20:26:29.436219 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.436204    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 20:26:29.438202 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.438181    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:29.463192 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.463173    5071 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 20:26:29.474632 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.474598    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 20:26:29.474763 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.474740    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:26:29.474891 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.474751    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 20:26:29.474891 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.474796    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 20:26:29.475000 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.474932    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:26:29.475000 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.474973    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:26:29.475098 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.475021    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 20:26:29.475098 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.475031    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 20:26:29.475196 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.475182    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 20:26:29.478164 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.478150    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 20:26:29.484236 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.484219    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 20:26:29.484323 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.484273    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:26:29.484554 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.484537    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 20:26:29.484912 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.484885    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 20:26:29.485075 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.484964    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 20:26:29.485075 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.485011    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ld6lr\""
Oct 30 20:26:29.485075 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.485047    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 20:26:29.485075 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.485056    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 20:26:29.485400 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.485148    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wczr5\""
Oct 30 20:26:29.485400 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.485203    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 20:26:29.485400 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.485211    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fngvs\""
Oct 30 20:26:29.485400 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.485308    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-vhzdq\""
Oct 30 20:26:29.485400 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.485371    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bx2zg\""
Oct 30 20:26:29.485579 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.485422    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-47trg\""
Oct 30 20:26:29.485579 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.485443    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 20:26:29.485579 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.485446    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-2t45q\""
Oct 30 20:26:29.485723 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.485704    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-cnj9l\""
Oct 30 20:26:29.485843 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.485738    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 20:26:29.486007 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.485987    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 20:26:29.486333 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.486285    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 20:26:29.486333 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.486317    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 20:26:29.486498 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.486363    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 20:26:29.486676 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.486660    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 20:26:29.486952 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.486934    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-5tplx\""
Oct 30 20:26:29.487152 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.486961    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lwgrd\""
Oct 30 20:26:29.487152 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.486984    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-qcj9t\""
Oct 30 20:26:29.487152 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.487121    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-mgc68\""
Oct 30 20:26:29.487509 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.487216    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mkv2p\""
Oct 30 20:26:29.487509 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.487276    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bxjhb\""
Oct 30 20:26:29.487509 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.487300    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9wdmm\""
Oct 30 20:26:29.487509 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.487331    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bpqv7\""
Oct 30 20:26:29.487509 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.487363    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 20:26:29.487509 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.487219    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qmclr\""
Oct 30 20:26:29.487509 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.487396    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-52gtb\""
Oct 30 20:26:29.487509 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.487410    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 20:26:29.487509 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.487473    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-78mnk\""
Oct 30 20:26:29.487879 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.487583    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 20:26:29.528076 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.528054    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 20:26:29.528209 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.528078    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 20:26:29.528209 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.528108    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 20:26:29.528209 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.528145    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 20:26:29.528369 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.528350    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 20:26:29.528549 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.528529    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 20:26:29.528549 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.528544    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 20:26:29.528736 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.528649    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 20:26:29.529483 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.529386    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:26:29.529987 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.529966    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 20:26:29.529987 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.529979    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:26:29.530313 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.530295    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 20:26:29.530390 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.530373    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 20:26:29.530582 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.530495    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 20:26:29.530935 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.530869    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 20:26:29.532310 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.531142    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 20:26:29.532310 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.531222    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 20:26:29.532310 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.531316    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 20:26:29.532310 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.531356    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 20:26:29.532310 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.531581    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 20:26:29.532310 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.531662    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 20:26:29.532310 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.531886    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 20:26:29.532310 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.531908    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 20:26:29.532687 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.532048    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 20:26:29.532687 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.532071    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 20:26:29.532877 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.532072    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 20:26:29.532877 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.532195    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 20:26:29.533022 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.532279    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 20:26:29.533121 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.532361    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 20:26:29.533171 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.532393    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 20:26:29.534125 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.532545    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 20:26:29.534125 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.533896    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 20:26:29.537128 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.537110    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 20:26:29.560272 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.560248    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f908129-4500-4433-871c-032c29a55dd9-trusted-ca\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:29.560398 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.560283    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70fa9600-c497-4deb-a133-3cb24f1c8050-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4v8mv\" (UID: \"70fa9600-c497-4deb-a133-3cb24f1c8050\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4v8mv"
Oct 30 20:26:29.560398 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.560300    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ee380d52-b71c-4839-8835-078d98a5e0ee-archives-path\") pod \"periodic-gathering-zmknv-22np8\" (UID: \"ee380d52-b71c-4839-8835-078d98a5e0ee\") " pod="openshift-insights/periodic-gathering-zmknv-22np8"
Oct 30 20:26:29.560398 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.560322    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-system-cni-dir\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.560398 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.560345    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-multus-daemon-config\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.560398 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.560380    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-var-lib-kubelet\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.560668 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.560478    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ee380d52-b71c-4839-8835-078d98a5e0ee-archives-path\") pod \"periodic-gathering-zmknv-22np8\" (UID: \"ee380d52-b71c-4839-8835-078d98a5e0ee\") " pod="openshift-insights/periodic-gathering-zmknv-22np8"
Oct 30 20:26:29.560668 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.560520    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-log-socket\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.560668 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.560543    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea45e6d2-1a47-4956-b652-19b428800f42-sys-fs\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:26:29.560668 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.560582    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8nqb\" (UniqueName: \"kubernetes.io/projected/78c7e317-9d58-4752-9804-92ac45b5afb7-kube-api-access-m8nqb\") pod \"kas-connection-checker-685895c7c9-c5b2k\" (UID: \"78c7e317-9d58-4752-9804-92ac45b5afb7\") " pod="kube-system/kas-connection-checker-685895c7c9-c5b2k"
Oct 30 20:26:29.560668 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.560616    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf-snapshots\") pod \"insights-operator-7c4986d449-mql5m\" (UID: \"8840f0ca-269e-4924-8b3d-5fe6ff4e90cf\") " pod="openshift-insights/insights-operator-7c4986d449-mql5m"
Oct 30 20:26:29.560668 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.560643    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-cni-binary-copy\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.560959 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.560669    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-metrics-certs\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:29.560959 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.560717    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-multus-daemon-config\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.560959 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.560929    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-cni-binary-copy\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.561095 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.560978    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/214f4f91-4d6e-46e0-ac1f-d23935cd7215-metrics-certs\") pod \"network-metrics-daemon-zcslg\" (UID: \"214f4f91-4d6e-46e0-ac1f-d23935cd7215\") " pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:26:29.561095 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.561018    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:26:29.561095 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.561046    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf-tmp\") pod \"insights-operator-7c4986d449-mql5m\" (UID: \"8840f0ca-269e-4924-8b3d-5fe6ff4e90cf\") " pod="openshift-insights/insights-operator-7c4986d449-mql5m"
Oct 30 20:26:29.561095 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.561069    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-hostroot\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.561095 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.561091    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f908129-4500-4433-871c-032c29a55dd9-trusted-ca\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:29.561360 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.561094    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-slash\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.561360 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.561133    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-run-ovn\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.561360 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.561160    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/80cae79e-b60f-4efe-bb58-e77753a457e5-ovn-node-metrics-cert\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.561360 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.561212    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/88d9611e-c7ed-490b-b401-530d1649c873-original-pull-secret\") pod \"global-pull-secret-syncer-bqh9l\" (UID: \"88d9611e-c7ed-490b-b401-530d1649c873\") " pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:26:29.561360 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.561237    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf-serving-cert\") pod \"insights-operator-7c4986d449-mql5m\" (UID: \"8840f0ca-269e-4924-8b3d-5fe6ff4e90cf\") " pod="openshift-insights/insights-operator-7c4986d449-mql5m"
Oct 30 20:26:29.561360 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.561262    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-multus-cni-dir\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.561360 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.561284    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-host-run-k8s-cni-cncf-io\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.561360 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.561343    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgf7l\" (UniqueName: \"kubernetes.io/projected/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-kube-api-access-zgf7l\") pod \"dns-default-h2vh7\" (UID: \"4c7e75b7-c3cd-4791-9d39-9120f6fd95dc\") " pod="openshift-dns/dns-default-h2vh7"
Oct 30 20:26:29.561704 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.561367    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-stats-auth\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:29.561704 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.561394    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w5vd\" (UniqueName: \"kubernetes.io/projected/88178646-9467-4225-b362-578f31454922-kube-api-access-2w5vd\") pod \"node-resolver-c8422\" (UID: \"88178646-9467-4225-b362-578f31454922\") " pod="openshift-dns/node-resolver-c8422"
Oct 30 20:26:29.561704 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.561417    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/05f22644-0884-434d-b56b-193d9554b505-installation-pull-secrets\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:29.561704 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.561438    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsmc8\" (UniqueName: \"kubernetes.io/projected/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf-kube-api-access-tsmc8\") pod \"insights-operator-7c4986d449-mql5m\" (UID: \"8840f0ca-269e-4924-8b3d-5fe6ff4e90cf\") " pod="openshift-insights/insights-operator-7c4986d449-mql5m"
Oct 30 20:26:29.561704 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.561460    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llcbk\" (UniqueName: \"kubernetes.io/projected/8b7865f8-cfd2-4938-8621-92b63faee7af-kube-api-access-llcbk\") pod \"node-ca-rdqrc\" (UID: \"8b7865f8-cfd2-4938-8621-92b63faee7af\") " pod="openshift-image-registry/node-ca-rdqrc"
Oct 30 20:26:29.561704 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.561480    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-system-cni-dir\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:26:29.561704 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.561554    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf-snapshots\") pod \"insights-operator-7c4986d449-mql5m\" (UID: \"8840f0ca-269e-4924-8b3d-5fe6ff4e90cf\") " pod="openshift-insights/insights-operator-7c4986d449-mql5m"
Oct 30 20:26:29.561704 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.561568    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnvlx\" (UniqueName: \"kubernetes.io/projected/309f81fb-9b40-4053-a060-8672a7c84f77-kube-api-access-tnvlx\") pod \"service-ca-5fffc97bdf-6b75b\" (UID: \"309f81fb-9b40-4053-a060-8672a7c84f77\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6b75b"
Oct 30 20:26:29.561704 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.561594    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-tuned\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.561704 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.561617    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwzfx\" (UniqueName: \"kubernetes.io/projected/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-kube-api-access-cwzfx\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.561704 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.561641    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsx89\" (UniqueName: \"kubernetes.io/projected/f3f7114f-f896-4fc9-b98a-e47574e7720d-kube-api-access-wsx89\") pod \"cluster-samples-operator-6ddb6c8b94-m9n59\" (UID: \"f3f7114f-f896-4fc9-b98a-e47574e7720d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-m9n59"
Oct 30 20:26:29.561704 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.561667    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ce6c7b90-6b06-4c81-bf95-539b71e2aaed-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf\" (UID: \"ce6c7b90-6b06-4c81-bf95-539b71e2aaed\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf"
Oct 30 20:26:29.561704 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.561688    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/89ed3d9f-e057-40a3-add4-d6374380be1f-agent-certs\") pod \"konnectivity-agent-4khkm\" (UID: \"89ed3d9f-e057-40a3-add4-d6374380be1f\") " pod="kube-system/konnectivity-agent-4khkm"
Oct 30 20:26:29.561704 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.561707    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/89ed3d9f-e057-40a3-add4-d6374380be1f-konnectivity-ca\") pod \"konnectivity-agent-4khkm\" (UID: \"89ed3d9f-e057-40a3-add4-d6374380be1f\") " pod="kube-system/konnectivity-agent-4khkm"
Oct 30 20:26:29.562320 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.561731    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b87eabe7-2403-4534-987b-ce6ecc6fda94-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tk6nb\" (UID: \"b87eabe7-2403-4534-987b-ce6ecc6fda94\") " pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:29.562320 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.561753    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b87eabe7-2403-4534-987b-ce6ecc6fda94-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tk6nb\" (UID: \"b87eabe7-2403-4534-987b-ce6ecc6fda94\") " pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:29.562320 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.561783    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzql8\" (UniqueName: \"kubernetes.io/projected/fc3bbdf6-9dad-4519-ae08-faf3cad565e2-kube-api-access-vzql8\") pod \"network-check-source-6f78d59f99-bhrtb\" (UID: \"fc3bbdf6-9dad-4519-ae08-faf3cad565e2\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bhrtb"
Oct 30 20:26:29.562320 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.561807    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-lib-modules\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.562320 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.561856    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-default-certificate\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:29.562320 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.561868    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:26:29.562320 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.562130    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-default-certificate\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:29.562320 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.562179    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/88d9611e-c7ed-490b-b401-530d1649c873-original-pull-secret\") pod \"global-pull-secret-syncer-bqh9l\" (UID: \"88d9611e-c7ed-490b-b401-530d1649c873\") " pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:26:29.562320 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.562296    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf-serving-cert\") pod \"insights-operator-7c4986d449-mql5m\" (UID: \"8840f0ca-269e-4924-8b3d-5fe6ff4e90cf\") " pod="openshift-insights/insights-operator-7c4986d449-mql5m"
Oct 30 20:26:29.562715 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.562327    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/80cae79e-b60f-4efe-bb58-e77753a457e5-ovn-node-metrics-cert\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.562715 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.562367    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1e8d4f3-e4a1-416c-bbc7-2c76063bfc24-cert\") pod \"ingress-canary-mrnvn\" (UID: \"c1e8d4f3-e4a1-416c-bbc7-2c76063bfc24\") " pod="openshift-ingress-canary/ingress-canary-mrnvn"
Oct 30 20:26:29.562715 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.562397    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-bound-sa-token\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:29.562715 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.562421    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/05f22644-0884-434d-b56b-193d9554b505-trusted-ca\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:29.562715 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.562446    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-cnibin\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:26:29.562715 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.562470    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b87eabe7-2403-4534-987b-ce6ecc6fda94-data-volume\") pod \"insights-runtime-extractor-tk6nb\" (UID: \"b87eabe7-2403-4534-987b-ce6ecc6fda94\") " pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:29.562715 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.562494    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-host-run-netns\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.562715 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.562521    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-multus-conf-dir\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.562715 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.562547    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/05f22644-0884-434d-b56b-193d9554b505-registry-certificates\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:29.562715 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.562572    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-host-var-lib-cni-bin\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.562715 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.562606    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea45e6d2-1a47-4956-b652-19b428800f42-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:26:29.562715 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.562634    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea45e6d2-1a47-4956-b652-19b428800f42-etc-selinux\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:26:29.562715 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.562664    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5zcp\" (UniqueName: \"kubernetes.io/projected/ea45e6d2-1a47-4956-b652-19b428800f42-kube-api-access-h5zcp\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:26:29.562715 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.562690    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-systemd\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.562715 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.562714    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-tmp\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.563427 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.562739    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-kubelet\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.563427 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.562745    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-stats-auth\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:29.563427 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.562764    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/05f22644-0884-434d-b56b-193d9554b505-ca-trust-extracted\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:29.563427 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.562793    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-cni-binary-copy\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:26:29.563427 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.562979    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-kubernetes\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.563427 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.563006    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-sysctl-conf\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.563427 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.563044    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b87eabe7-2403-4534-987b-ce6ecc6fda94-crio-socket\") pod \"insights-runtime-extractor-tk6nb\" (UID: \"b87eabe7-2403-4534-987b-ce6ecc6fda94\") " pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:29.563427 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.563073    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/309f81fb-9b40-4053-a060-8672a7c84f77-signing-cabundle\") pod \"service-ca-5fffc97bdf-6b75b\" (UID: \"309f81fb-9b40-4053-a060-8672a7c84f77\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6b75b"
Oct 30 20:26:29.563427 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.563088    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf-tmp\") pod \"insights-operator-7c4986d449-mql5m\" (UID: \"8840f0ca-269e-4924-8b3d-5fe6ff4e90cf\") " pod="openshift-insights/insights-operator-7c4986d449-mql5m"
Oct 30 20:26:29.563427 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.563101    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee380d52-b71c-4839-8835-078d98a5e0ee-trusted-ca-bundle\") pod \"periodic-gathering-zmknv-22np8\" (UID: \"ee380d52-b71c-4839-8835-078d98a5e0ee\") " pod="openshift-insights/periodic-gathering-zmknv-22np8"
Oct 30 20:26:29.563427 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.563129    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss27z\" (UniqueName: \"kubernetes.io/projected/50f563bf-3336-49c6-9708-041c41b41c53-kube-api-access-ss27z\") pod \"migrator-66df5d496b-hbm7v\" (UID: \"50f563bf-3336-49c6-9708-041c41b41c53\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hbm7v"
Oct 30 20:26:29.563427 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.563156    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/88d9611e-c7ed-490b-b401-530d1649c873-kubelet-config\") pod \"global-pull-secret-syncer-bqh9l\" (UID: \"88d9611e-c7ed-490b-b401-530d1649c873\") " pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:26:29.563427 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.563184    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-registry-tls\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:29.563427 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.563199    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-tuned\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.563427 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.563208    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d5312b70-82df-4e75-a300-b8e1d63a75f3-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-9rs22\" (UID: \"d5312b70-82df-4e75-a300-b8e1d63a75f3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9rs22"
Oct 30 20:26:29.563427 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.563260    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-host-var-lib-kubelet\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.563427 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.563299    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkqqp\" (UniqueName: \"kubernetes.io/projected/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-kube-api-access-mkqqp\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.563427 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.563327    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea45e6d2-1a47-4956-b652-19b428800f42-socket-dir\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:26:29.563427 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.563336    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/89ed3d9f-e057-40a3-add4-d6374380be1f-konnectivity-ca\") pod \"konnectivity-agent-4khkm\" (UID: \"89ed3d9f-e057-40a3-add4-d6374380be1f\") " pod="kube-system/konnectivity-agent-4khkm"
Oct 30 20:26:29.563427 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.563352    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-systemd-units\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.563427 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.563382    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dx86\" (UniqueName: \"kubernetes.io/projected/a0790a28-d605-4c0c-a652-41ac6ef22e09-kube-api-access-4dx86\") pod \"volume-data-source-validator-58c9f5f46d-z9tdd\" (UID: \"a0790a28-d605-4c0c-a652-41ac6ef22e09\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z9tdd"
Oct 30 20:26:29.563427 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.563411    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee380d52-b71c-4839-8835-078d98a5e0ee-service-ca-bundle\") pod \"periodic-gathering-zmknv-22np8\" (UID: \"ee380d52-b71c-4839-8835-078d98a5e0ee\") " pod="openshift-insights/periodic-gathering-zmknv-22np8"
Oct 30 20:26:29.563427 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.563437    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-host-run-multus-certs\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.564455 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.563470    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/91fe9e06-0c30-4000-bddb-2ba89cd048cc-serving-cert\") pod \"console-operator-7cd56b8dbc-l8qvb\" (UID: \"91fe9e06-0c30-4000-bddb-2ba89cd048cc\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb"
Oct 30 20:26:29.564455 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.563500    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf-service-ca-bundle\") pod \"insights-operator-7c4986d449-mql5m\" (UID: \"8840f0ca-269e-4924-8b3d-5fe6ff4e90cf\") " pod="openshift-insights/insights-operator-7c4986d449-mql5m"
Oct 30 20:26:29.564455 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.563526    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-modprobe-d\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.564455 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.563551    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f3f7114f-f896-4fc9-b98a-e47574e7720d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-m9n59\" (UID: \"f3f7114f-f896-4fc9-b98a-e47574e7720d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-m9n59"
Oct 30 20:26:29.564455 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.563567    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-cni-binary-copy\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:26:29.564455 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.563761    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/89ed3d9f-e057-40a3-add4-d6374380be1f-agent-certs\") pod \"konnectivity-agent-4khkm\" (UID: \"89ed3d9f-e057-40a3-add4-d6374380be1f\") " pod="kube-system/konnectivity-agent-4khkm"
Oct 30 20:26:29.564455 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.563412    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d5312b70-82df-4e75-a300-b8e1d63a75f3-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-9rs22\" (UID: \"d5312b70-82df-4e75-a300-b8e1d63a75f3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9rs22"
Oct 30 20:26:29.564455 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.563923    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ce6c7b90-6b06-4c81-bf95-539b71e2aaed-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf\" (UID: \"ce6c7b90-6b06-4c81-bf95-539b71e2aaed\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf"
Oct 30 20:26:29.564455 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.563994    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/05f22644-0884-434d-b56b-193d9554b505-installation-pull-secrets\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:29.564455 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.564057    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-run-systemd\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.564455 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.564091    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-node-log\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.564455 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.564121    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/111798c9-6959-4dcb-9a8a-db3037f7d662-iptables-alerter-script\") pod \"iptables-alerter-kwm7g\" (UID: \"111798c9-6959-4dcb-9a8a-db3037f7d662\") " pod="openshift-network-operator/iptables-alerter-kwm7g"
Oct 30 20:26:29.564455 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.564155    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgvld\" (UniqueName: \"kubernetes.io/projected/111798c9-6959-4dcb-9a8a-db3037f7d662-kube-api-access-pgvld\") pod \"iptables-alerter-kwm7g\" (UID: \"111798c9-6959-4dcb-9a8a-db3037f7d662\") " pod="openshift-network-operator/iptables-alerter-kwm7g"
Oct 30 20:26:29.564455 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.564184    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8b7865f8-cfd2-4938-8621-92b63faee7af-serviceca\") pod \"node-ca-rdqrc\" (UID: \"8b7865f8-cfd2-4938-8621-92b63faee7af\") " pod="openshift-image-registry/node-ca-rdqrc"
Oct 30 20:26:29.564455 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.564214    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wb98\" (UniqueName: \"kubernetes.io/projected/d5312b70-82df-4e75-a300-b8e1d63a75f3-kube-api-access-5wb98\") pod \"cluster-monitoring-operator-884576c57-9rs22\" (UID: \"d5312b70-82df-4e75-a300-b8e1d63a75f3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9rs22"
Oct 30 20:26:29.564455 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.564241    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-cnibin\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.564455 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.564279    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-sysctl-d\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.564455 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.564310    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-var-lib-openvswitch\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.564455 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.564337    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fxk9\" (UniqueName: \"kubernetes.io/projected/c1e8d4f3-e4a1-416c-bbc7-2c76063bfc24-kube-api-access-7fxk9\") pod \"ingress-canary-mrnvn\" (UID: \"c1e8d4f3-e4a1-416c-bbc7-2c76063bfc24\") " pod="openshift-ingress-canary/ingress-canary-mrnvn"
Oct 30 20:26:29.564455 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.564364    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/309f81fb-9b40-4053-a060-8672a7c84f77-signing-key\") pod \"service-ca-5fffc97bdf-6b75b\" (UID: \"309f81fb-9b40-4053-a060-8672a7c84f77\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6b75b"
Oct 30 20:26:29.564455 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.564387    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-run\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.564455 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.564390    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b87eabe7-2403-4534-987b-ce6ecc6fda94-data-volume\") pod \"insights-runtime-extractor-tk6nb\" (UID: \"b87eabe7-2403-4534-987b-ce6ecc6fda94\") " pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:29.564455 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.564412    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-run-netns\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.564455 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.564439    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-bound-sa-token\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:29.565841 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.564467    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh4r9\" (UniqueName: \"kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-kube-api-access-qh4r9\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:29.565841 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.564499    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/91fe9e06-0c30-4000-bddb-2ba89cd048cc-trusted-ca\") pod \"console-operator-7cd56b8dbc-l8qvb\" (UID: \"91fe9e06-0c30-4000-bddb-2ba89cd048cc\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb"
Oct 30 20:26:29.565841 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.564525    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-sys\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.565841 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.564531    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/05f22644-0884-434d-b56b-193d9554b505-registry-certificates\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:29.565841 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.564556    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-etc-openvswitch\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.565841 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.564588    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-tmp\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.565841 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.564649    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1e8d4f3-e4a1-416c-bbc7-2c76063bfc24-cert\") pod \"ingress-canary-mrnvn\" (UID: \"c1e8d4f3-e4a1-416c-bbc7-2c76063bfc24\") " pod="openshift-ingress-canary/ingress-canary-mrnvn"
Oct 30 20:26:29.565841 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.564650    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/05f22644-0884-434d-b56b-193d9554b505-ca-trust-extracted\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:29.565841 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.564787    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8b7865f8-cfd2-4938-8621-92b63faee7af-serviceca\") pod \"node-ca-rdqrc\" (UID: \"8b7865f8-cfd2-4938-8621-92b63faee7af\") " pod="openshift-image-registry/node-ca-rdqrc"
Oct 30 20:26:29.565841 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.565041    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee380d52-b71c-4839-8835-078d98a5e0ee-service-ca-bundle\") pod \"periodic-gathering-zmknv-22np8\" (UID: \"ee380d52-b71c-4839-8835-078d98a5e0ee\") " pod="openshift-insights/periodic-gathering-zmknv-22np8"
Oct 30 20:26:29.565841 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.565163    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf-service-ca-bundle\") pod \"insights-operator-7c4986d449-mql5m\" (UID: \"8840f0ca-269e-4924-8b3d-5fe6ff4e90cf\") " pod="openshift-insights/insights-operator-7c4986d449-mql5m"
Oct 30 20:26:29.565841 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.565201    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/309f81fb-9b40-4053-a060-8672a7c84f77-signing-key\") pod \"service-ca-5fffc97bdf-6b75b\" (UID: \"309f81fb-9b40-4053-a060-8672a7c84f77\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6b75b"
Oct 30 20:26:29.565841 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.565285    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/91fe9e06-0c30-4000-bddb-2ba89cd048cc-serving-cert\") pod \"console-operator-7cd56b8dbc-l8qvb\" (UID: \"91fe9e06-0c30-4000-bddb-2ba89cd048cc\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb"
Oct 30 20:26:29.565841 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.565391    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/05f22644-0884-434d-b56b-193d9554b505-trusted-ca\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:29.565841 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.565629    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/309f81fb-9b40-4053-a060-8672a7c84f77-signing-cabundle\") pod \"service-ca-5fffc97bdf-6b75b\" (UID: \"309f81fb-9b40-4053-a060-8672a7c84f77\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6b75b"
Oct 30 20:26:29.565841 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.565674    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-run-ovn-kubernetes\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.565841 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.565709    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/80cae79e-b60f-4efe-bb58-e77753a457e5-env-overrides\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.565841 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.565739    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64rvp\" (UniqueName: \"kubernetes.io/projected/5040db9b-91d3-4d89-adaa-d1394cc3a617-kube-api-access-64rvp\") pod \"service-ca-operator-544d749c8c-tp7sc\" (UID: \"5040db9b-91d3-4d89-adaa-d1394cc3a617\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tp7sc"
Oct 30 20:26:29.565841 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.565772    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-cni-netd\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.565841 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.565836    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee380d52-b71c-4839-8835-078d98a5e0ee-trusted-ca-bundle\") pod \"periodic-gathering-zmknv-22np8\" (UID: \"ee380d52-b71c-4839-8835-078d98a5e0ee\") " pod="openshift-insights/periodic-gathering-zmknv-22np8"
Oct 30 20:26:29.566698 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.565859    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/111798c9-6959-4dcb-9a8a-db3037f7d662-iptables-alerter-script\") pod \"iptables-alerter-kwm7g\" (UID: \"111798c9-6959-4dcb-9a8a-db3037f7d662\") " pod="openshift-network-operator/iptables-alerter-kwm7g"
Oct 30 20:26:29.566698 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.565868    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/80cae79e-b60f-4efe-bb58-e77753a457e5-ovnkube-config\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.566698 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.565900    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/70fa9600-c497-4deb-a133-3cb24f1c8050-nginx-conf\") pod \"networking-console-plugin-86c478bb88-4v8mv\" (UID: \"70fa9600-c497-4deb-a133-3cb24f1c8050\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4v8mv"
Oct 30 20:26:29.566698 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.565928    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-etc-kubernetes\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.566698 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.565954    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/111798c9-6959-4dcb-9a8a-db3037f7d662-host-slash\") pod \"iptables-alerter-kwm7g\" (UID: \"111798c9-6959-4dcb-9a8a-db3037f7d662\") " pod="openshift-network-operator/iptables-alerter-kwm7g"
Oct 30 20:26:29.566698 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.565982    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/80cae79e-b60f-4efe-bb58-e77753a457e5-env-overrides\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.566698 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.566025    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krgxb\" (UniqueName: \"kubernetes.io/projected/533fb89e-0b69-4c79-9cbf-7baa5f725219-kube-api-access-krgxb\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:29.566698 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.566057    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88178646-9467-4225-b362-578f31454922-tmp-dir\") pod \"node-resolver-c8422\" (UID: \"88178646-9467-4225-b362-578f31454922\") " pod="openshift-dns/node-resolver-c8422"
Oct 30 20:26:29.566698 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.566081    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d5312b70-82df-4e75-a300-b8e1d63a75f3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9rs22\" (UID: \"d5312b70-82df-4e75-a300-b8e1d63a75f3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9rs22"
Oct 30 20:26:29.566698 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.566108    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-sysconfig\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.566698 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.566136    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv49q\" (UniqueName: \"kubernetes.io/projected/08b2f787-ad25-47f9-8ef8-724308458f88-kube-api-access-hv49q\") pod \"kas-connection-checker-685895c7c9-brf75\" (UID: \"08b2f787-ad25-47f9-8ef8-724308458f88\") " pod="kube-system/kas-connection-checker-685895c7c9-brf75"
Oct 30 20:26:29.566698 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.566217    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88178646-9467-4225-b362-578f31454922-tmp-dir\") pod \"node-resolver-c8422\" (UID: \"88178646-9467-4225-b362-578f31454922\") " pod="openshift-dns/node-resolver-c8422"
Oct 30 20:26:29.566698 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.566218    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/80cae79e-b60f-4efe-bb58-e77753a457e5-ovnkube-config\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.566698 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.566252    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9f908129-4500-4433-871c-032c29a55dd9-ca-trust-extracted\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:29.566698 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.566288    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9f908129-4500-4433-871c-032c29a55dd9-registry-certificates\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:29.566698 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.566318    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2gcf\" (UniqueName: \"kubernetes.io/projected/b87eabe7-2403-4534-987b-ce6ecc6fda94-kube-api-access-g2gcf\") pod \"insights-runtime-extractor-tk6nb\" (UID: \"b87eabe7-2403-4534-987b-ce6ecc6fda94\") " pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:29.566698 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.566348    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/533fb89e-0b69-4c79-9cbf-7baa5f725219-service-ca-bundle\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:29.566698 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.566379    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/80cae79e-b60f-4efe-bb58-e77753a457e5-ovnkube-script-lib\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.566698 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.566407    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-run-openvswitch\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.566698 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.566602    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9f908129-4500-4433-871c-032c29a55dd9-registry-certificates\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:29.566698 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.566612    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/91fe9e06-0c30-4000-bddb-2ba89cd048cc-trusted-ca\") pod \"console-operator-7cd56b8dbc-l8qvb\" (UID: \"91fe9e06-0c30-4000-bddb-2ba89cd048cc\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb"
Oct 30 20:26:29.566698 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.566660    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjn87\" (UniqueName: \"kubernetes.io/projected/80cae79e-b60f-4efe-bb58-e77753a457e5-kube-api-access-kjn87\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.566698 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.566693    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tpmm\" (UniqueName: \"kubernetes.io/projected/ce6c7b90-6b06-4c81-bf95-539b71e2aaed-kube-api-access-8tpmm\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf\" (UID: \"ce6c7b90-6b06-4c81-bf95-539b71e2aaed\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf"
Oct 30 20:26:29.567719 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.566751    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6kxw\" (UniqueName: \"kubernetes.io/projected/fe578b91-829c-450a-a60a-0f4e4b6e8c63-kube-api-access-m6kxw\") pod \"migrator-66df5d496b-lfrbc\" (UID: \"fe578b91-829c-450a-a60a-0f4e4b6e8c63\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lfrbc"
Oct 30 20:26:29.567719 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.566785    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9f908129-4500-4433-871c-032c29a55dd9-installation-pull-secrets\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:29.567719 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.566800    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/80cae79e-b60f-4efe-bb58-e77753a457e5-ovnkube-script-lib\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.567719 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.566839    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb792\" (UniqueName: \"kubernetes.io/projected/c9732ee5-c72e-44c6-9917-49cc9f81ffe0-kube-api-access-bb792\") pod \"network-check-target-pfvnn\" (UID: \"c9732ee5-c72e-44c6-9917-49cc9f81ffe0\") " pod="openshift-network-diagnostics/network-check-target-pfvnn"
Oct 30 20:26:29.567719 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.566880    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx2vz\" (UniqueName: \"kubernetes.io/projected/214f4f91-4d6e-46e0-ac1f-d23935cd7215-kube-api-access-nx2vz\") pod \"network-metrics-daemon-zcslg\" (UID: \"214f4f91-4d6e-46e0-ac1f-d23935cd7215\") " pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:26:29.567719 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.566785    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/70fa9600-c497-4deb-a133-3cb24f1c8050-nginx-conf\") pod \"networking-console-plugin-86c478bb88-4v8mv\" (UID: \"70fa9600-c497-4deb-a133-3cb24f1c8050\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4v8mv"
Oct 30 20:26:29.567719 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.566974    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/88178646-9467-4225-b362-578f31454922-hosts-file\") pod \"node-resolver-c8422\" (UID: \"88178646-9467-4225-b362-578f31454922\") " pod="openshift-dns/node-resolver-c8422"
Oct 30 20:26:29.567719 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.567012    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94tlh\" (UniqueName: \"kubernetes.io/projected/91fe9e06-0c30-4000-bddb-2ba89cd048cc-kube-api-access-94tlh\") pod \"console-operator-7cd56b8dbc-l8qvb\" (UID: \"91fe9e06-0c30-4000-bddb-2ba89cd048cc\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb"
Oct 30 20:26:29.567719 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.567045    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:26:29.567719 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.567054    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9f908129-4500-4433-871c-032c29a55dd9-ca-trust-extracted\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:29.567719 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.567072    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea45e6d2-1a47-4956-b652-19b428800f42-registration-dir\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:26:29.567719 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.567347    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-host\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.567719 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.567380    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9f908129-4500-4433-871c-032c29a55dd9-installation-pull-secrets\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:29.567719 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.567404    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-metrics-tls\") pod \"dns-default-h2vh7\" (UID: \"4c7e75b7-c3cd-4791-9d39-9120f6fd95dc\") " pod="openshift-dns/dns-default-h2vh7"
Oct 30 20:26:29.567719 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.567465    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8hdc\" (UniqueName: \"kubernetes.io/projected/f4913de8-6db5-413a-938f-abc50a5460e7-kube-api-access-d8hdc\") pod \"kas-connection-checker-685895c7c9-kgsjn\" (UID: \"f4913de8-6db5-413a-938f-abc50a5460e7\") " pod="kube-system/kas-connection-checker-685895c7c9-kgsjn"
Oct 30 20:26:29.567719 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.567513    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-registry-tls\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:29.567719 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.567583    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5040db9b-91d3-4d89-adaa-d1394cc3a617-config\") pod \"service-ca-operator-544d749c8c-tp7sc\" (UID: \"5040db9b-91d3-4d89-adaa-d1394cc3a617\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tp7sc"
Oct 30 20:26:29.567719 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.567616    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:26:29.567719 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.567636    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-multus-socket-dir-parent\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.567719 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.567675    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-host-var-lib-cni-multus\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.569085 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.567808    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:26:29.569085 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.567843    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5040db9b-91d3-4d89-adaa-d1394cc3a617-config\") pod \"service-ca-operator-544d749c8c-tp7sc\" (UID: \"5040db9b-91d3-4d89-adaa-d1394cc3a617\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tp7sc"
Oct 30 20:26:29.569085 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.567914    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-config-volume\") pod \"dns-default-h2vh7\" (UID: \"4c7e75b7-c3cd-4791-9d39-9120f6fd95dc\") " pod="openshift-dns/dns-default-h2vh7"
Oct 30 20:26:29.569085 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.567951    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-tmp-dir\") pod \"dns-default-h2vh7\" (UID: \"4c7e75b7-c3cd-4791-9d39-9120f6fd95dc\") " pod="openshift-dns/dns-default-h2vh7"
Oct 30 20:26:29.569085 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.568005    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ce6c7b90-6b06-4c81-bf95-539b71e2aaed-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf\" (UID: \"ce6c7b90-6b06-4c81-bf95-539b71e2aaed\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf"
Oct 30 20:26:29.569085 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.568049    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-tmp-dir\") pod \"dns-default-h2vh7\" (UID: \"4c7e75b7-c3cd-4791-9d39-9120f6fd95dc\") " pod="openshift-dns/dns-default-h2vh7"
Oct 30 20:26:29.569085 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.568104    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9f908129-4500-4433-871c-032c29a55dd9-image-registry-private-configuration\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:29.569085 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.568135    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b7865f8-cfd2-4938-8621-92b63faee7af-host\") pod \"node-ca-rdqrc\" (UID: \"8b7865f8-cfd2-4938-8621-92b63faee7af\") " pod="openshift-image-registry/node-ca-rdqrc"
Oct 30 20:26:29.569085 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.568136    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-config-volume\") pod \"dns-default-h2vh7\" (UID: \"4c7e75b7-c3cd-4791-9d39-9120f6fd95dc\") " pod="openshift-dns/dns-default-h2vh7"
Oct 30 20:26:29.569085 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.568164    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/91fe9e06-0c30-4000-bddb-2ba89cd048cc-config\") pod \"console-operator-7cd56b8dbc-l8qvb\" (UID: \"91fe9e06-0c30-4000-bddb-2ba89cd048cc\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb"
Oct 30 20:26:29.569085 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.568281    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9f908129-4500-4433-871c-032c29a55dd9-image-registry-private-configuration\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:29.569085 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.568315    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/05f22644-0884-434d-b56b-193d9554b505-image-registry-private-configuration\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:29.569085 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.568337    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l2sr\" (UniqueName: \"kubernetes.io/projected/ee380d52-b71c-4839-8835-078d98a5e0ee-kube-api-access-9l2sr\") pod \"periodic-gathering-zmknv-22np8\" (UID: \"ee380d52-b71c-4839-8835-078d98a5e0ee\") " pod="openshift-insights/periodic-gathering-zmknv-22np8"
Oct 30 20:26:29.569085 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.568356    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea45e6d2-1a47-4956-b652-19b428800f42-device-dir\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:26:29.569085 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.568395    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ce6c7b90-6b06-4c81-bf95-539b71e2aaed-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf\" (UID: \"ce6c7b90-6b06-4c81-bf95-539b71e2aaed\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf"
Oct 30 20:26:29.569085 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.568399    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.569085 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.568441    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqftg\" (UniqueName: \"kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-kube-api-access-nqftg\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:29.569085 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.568502    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f3f7114f-f896-4fc9-b98a-e47574e7720d-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-m9n59\" (UID: \"f3f7114f-f896-4fc9-b98a-e47574e7720d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-m9n59"
Oct 30 20:26:29.569085 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.568511    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/05f22644-0884-434d-b56b-193d9554b505-image-registry-private-configuration\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:29.569085 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.568531    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-cni-bin\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.569085 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.568659    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4vlt\" (UniqueName: \"kubernetes.io/projected/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-kube-api-access-l4vlt\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:26:29.569085 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.568696    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5040db9b-91d3-4d89-adaa-d1394cc3a617-serving-cert\") pod \"service-ca-operator-544d749c8c-tp7sc\" (UID: \"5040db9b-91d3-4d89-adaa-d1394cc3a617\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tp7sc"
Oct 30 20:26:29.569085 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.568725    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/88d9611e-c7ed-490b-b401-530d1649c873-dbus\") pod \"global-pull-secret-syncer-bqh9l\" (UID: \"88d9611e-c7ed-490b-b401-530d1649c873\") " pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:26:29.569085 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.568772    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/91fe9e06-0c30-4000-bddb-2ba89cd048cc-config\") pod \"console-operator-7cd56b8dbc-l8qvb\" (UID: \"91fe9e06-0c30-4000-bddb-2ba89cd048cc\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb"
Oct 30 20:26:29.569085 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.568790    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-mql5m\" (UID: \"8840f0ca-269e-4924-8b3d-5fe6ff4e90cf\") " pod="openshift-insights/insights-operator-7c4986d449-mql5m"
Oct 30 20:26:29.570153 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.569190    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f3f7114f-f896-4fc9-b98a-e47574e7720d-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-m9n59\" (UID: \"f3f7114f-f896-4fc9-b98a-e47574e7720d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-m9n59"
Oct 30 20:26:29.570153 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.569351    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5040db9b-91d3-4d89-adaa-d1394cc3a617-serving-cert\") pod \"service-ca-operator-544d749c8c-tp7sc\" (UID: \"5040db9b-91d3-4d89-adaa-d1394cc3a617\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tp7sc"
Oct 30 20:26:29.570153 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.569400    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-mql5m\" (UID: \"8840f0ca-269e-4924-8b3d-5fe6ff4e90cf\") " pod="openshift-insights/insights-operator-7c4986d449-mql5m"
Oct 30 20:26:29.610681 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.610611    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:29.619237 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.619207    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w5vd\" (UniqueName: \"kubernetes.io/projected/88178646-9467-4225-b362-578f31454922-kube-api-access-2w5vd\") pod \"node-resolver-c8422\" (UID: \"88178646-9467-4225-b362-578f31454922\") " pod="openshift-dns/node-resolver-c8422"
Oct 30 20:26:29.622353 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.621683    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzql8\" (UniqueName: \"kubernetes.io/projected/fc3bbdf6-9dad-4519-ae08-faf3cad565e2-kube-api-access-vzql8\") pod \"network-check-source-6f78d59f99-bhrtb\" (UID: \"fc3bbdf6-9dad-4519-ae08-faf3cad565e2\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bhrtb"
Oct 30 20:26:29.622353 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.621772    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5zcp\" (UniqueName: \"kubernetes.io/projected/ea45e6d2-1a47-4956-b652-19b428800f42-kube-api-access-h5zcp\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:26:29.622353 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.621882    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8nqb\" (UniqueName: \"kubernetes.io/projected/78c7e317-9d58-4752-9804-92ac45b5afb7-kube-api-access-m8nqb\") pod \"kas-connection-checker-685895c7c9-c5b2k\" (UID: \"78c7e317-9d58-4752-9804-92ac45b5afb7\") " pod="kube-system/kas-connection-checker-685895c7c9-c5b2k"
Oct 30 20:26:29.622353 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.621942    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkqqp\" (UniqueName: \"kubernetes.io/projected/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-kube-api-access-mkqqp\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.622353 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.622120    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llcbk\" (UniqueName: \"kubernetes.io/projected/8b7865f8-cfd2-4938-8621-92b63faee7af-kube-api-access-llcbk\") pod \"node-ca-rdqrc\" (UID: \"8b7865f8-cfd2-4938-8621-92b63faee7af\") " pod="openshift-image-registry/node-ca-rdqrc"
Oct 30 20:26:29.622353 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.622291    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsmc8\" (UniqueName: \"kubernetes.io/projected/8840f0ca-269e-4924-8b3d-5fe6ff4e90cf-kube-api-access-tsmc8\") pod \"insights-operator-7c4986d449-mql5m\" (UID: \"8840f0ca-269e-4924-8b3d-5fe6ff4e90cf\") " pod="openshift-insights/insights-operator-7c4986d449-mql5m"
Oct 30 20:26:29.622353 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.622302    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgf7l\" (UniqueName: \"kubernetes.io/projected/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-kube-api-access-zgf7l\") pod \"dns-default-h2vh7\" (UID: \"4c7e75b7-c3cd-4791-9d39-9120f6fd95dc\") " pod="openshift-dns/dns-default-h2vh7"
Oct 30 20:26:29.622353 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.622324    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnvlx\" (UniqueName: \"kubernetes.io/projected/309f81fb-9b40-4053-a060-8672a7c84f77-kube-api-access-tnvlx\") pod \"service-ca-5fffc97bdf-6b75b\" (UID: \"309f81fb-9b40-4053-a060-8672a7c84f77\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6b75b"
Oct 30 20:26:29.622700 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.622494    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsx89\" (UniqueName: \"kubernetes.io/projected/f3f7114f-f896-4fc9-b98a-e47574e7720d-kube-api-access-wsx89\") pod \"cluster-samples-operator-6ddb6c8b94-m9n59\" (UID: \"f3f7114f-f896-4fc9-b98a-e47574e7720d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-m9n59"
Oct 30 20:26:29.623420 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.623393    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss27z\" (UniqueName: \"kubernetes.io/projected/50f563bf-3336-49c6-9708-041c41b41c53-kube-api-access-ss27z\") pod \"migrator-66df5d496b-hbm7v\" (UID: \"50f563bf-3336-49c6-9708-041c41b41c53\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hbm7v"
Oct 30 20:26:29.623569 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.623552    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-bound-sa-token\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:29.623700 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.623681    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwzfx\" (UniqueName: \"kubernetes.io/projected/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-kube-api-access-cwzfx\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.623926 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.623903    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wb98\" (UniqueName: \"kubernetes.io/projected/d5312b70-82df-4e75-a300-b8e1d63a75f3-kube-api-access-5wb98\") pod \"cluster-monitoring-operator-884576c57-9rs22\" (UID: \"d5312b70-82df-4e75-a300-b8e1d63a75f3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9rs22"
Oct 30 20:26:29.624990 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.624266    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh4r9\" (UniqueName: \"kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-kube-api-access-qh4r9\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:29.624990 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.624326    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64rvp\" (UniqueName: \"kubernetes.io/projected/5040db9b-91d3-4d89-adaa-d1394cc3a617-kube-api-access-64rvp\") pod \"service-ca-operator-544d749c8c-tp7sc\" (UID: \"5040db9b-91d3-4d89-adaa-d1394cc3a617\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tp7sc"
Oct 30 20:26:29.624990 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.624541    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dx86\" (UniqueName: \"kubernetes.io/projected/a0790a28-d605-4c0c-a652-41ac6ef22e09-kube-api-access-4dx86\") pod \"volume-data-source-validator-58c9f5f46d-z9tdd\" (UID: \"a0790a28-d605-4c0c-a652-41ac6ef22e09\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z9tdd"
Oct 30 20:26:29.624990 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.624745    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgvld\" (UniqueName: \"kubernetes.io/projected/111798c9-6959-4dcb-9a8a-db3037f7d662-kube-api-access-pgvld\") pod \"iptables-alerter-kwm7g\" (UID: \"111798c9-6959-4dcb-9a8a-db3037f7d662\") " pod="openshift-network-operator/iptables-alerter-kwm7g"
Oct 30 20:26:29.624990 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.624852    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv49q\" (UniqueName: \"kubernetes.io/projected/08b2f787-ad25-47f9-8ef8-724308458f88-kube-api-access-hv49q\") pod \"kas-connection-checker-685895c7c9-brf75\" (UID: \"08b2f787-ad25-47f9-8ef8-724308458f88\") " pod="kube-system/kas-connection-checker-685895c7c9-brf75"
Oct 30 20:26:29.624990 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.624874    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-bound-sa-token\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:29.624990 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.624941    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fxk9\" (UniqueName: \"kubernetes.io/projected/c1e8d4f3-e4a1-416c-bbc7-2c76063bfc24-kube-api-access-7fxk9\") pod \"ingress-canary-mrnvn\" (UID: \"c1e8d4f3-e4a1-416c-bbc7-2c76063bfc24\") " pod="openshift-ingress-canary/ingress-canary-mrnvn"
Oct 30 20:26:29.625256 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.625006    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krgxb\" (UniqueName: \"kubernetes.io/projected/533fb89e-0b69-4c79-9cbf-7baa5f725219-kube-api-access-krgxb\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:29.625731 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.625710    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2gcf\" (UniqueName: \"kubernetes.io/projected/b87eabe7-2403-4534-987b-ce6ecc6fda94-kube-api-access-g2gcf\") pod \"insights-runtime-extractor-tk6nb\" (UID: \"b87eabe7-2403-4534-987b-ce6ecc6fda94\") " pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:29.625803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.625728    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l2sr\" (UniqueName: \"kubernetes.io/projected/ee380d52-b71c-4839-8835-078d98a5e0ee-kube-api-access-9l2sr\") pod \"periodic-gathering-zmknv-22np8\" (UID: \"ee380d52-b71c-4839-8835-078d98a5e0ee\") " pod="openshift-insights/periodic-gathering-zmknv-22np8"
Oct 30 20:26:29.625803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.625710    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjn87\" (UniqueName: \"kubernetes.io/projected/80cae79e-b60f-4efe-bb58-e77753a457e5-kube-api-access-kjn87\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.625803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.625713    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx2vz\" (UniqueName: \"kubernetes.io/projected/214f4f91-4d6e-46e0-ac1f-d23935cd7215-kube-api-access-nx2vz\") pod \"network-metrics-daemon-zcslg\" (UID: \"214f4f91-4d6e-46e0-ac1f-d23935cd7215\") " pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:26:29.626299 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.626256    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8hdc\" (UniqueName: \"kubernetes.io/projected/f4913de8-6db5-413a-938f-abc50a5460e7-kube-api-access-d8hdc\") pod \"kas-connection-checker-685895c7c9-kgsjn\" (UID: \"f4913de8-6db5-413a-938f-abc50a5460e7\") " pod="kube-system/kas-connection-checker-685895c7c9-kgsjn"
Oct 30 20:26:29.626423 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.626363    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tpmm\" (UniqueName: \"kubernetes.io/projected/ce6c7b90-6b06-4c81-bf95-539b71e2aaed-kube-api-access-8tpmm\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf\" (UID: \"ce6c7b90-6b06-4c81-bf95-539b71e2aaed\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf"
Oct 30 20:26:29.626493 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.626422    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94tlh\" (UniqueName: \"kubernetes.io/projected/91fe9e06-0c30-4000-bddb-2ba89cd048cc-kube-api-access-94tlh\") pod \"console-operator-7cd56b8dbc-l8qvb\" (UID: \"91fe9e06-0c30-4000-bddb-2ba89cd048cc\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb"
Oct 30 20:26:29.626852 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.626796    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6kxw\" (UniqueName: \"kubernetes.io/projected/fe578b91-829c-450a-a60a-0f4e4b6e8c63-kube-api-access-m6kxw\") pod \"migrator-66df5d496b-lfrbc\" (UID: \"fe578b91-829c-450a-a60a-0f4e4b6e8c63\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lfrbc"
Oct 30 20:26:29.626947 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.626857    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4vlt\" (UniqueName: \"kubernetes.io/projected/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-kube-api-access-l4vlt\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:26:29.627007 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.626994    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb792\" (UniqueName: \"kubernetes.io/projected/c9732ee5-c72e-44c6-9917-49cc9f81ffe0-kube-api-access-bb792\") pod \"network-check-target-pfvnn\" (UID: \"c9732ee5-c72e-44c6-9917-49cc9f81ffe0\") " pod="openshift-network-diagnostics/network-check-target-pfvnn"
Oct 30 20:26:29.627493 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.627466    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqftg\" (UniqueName: \"kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-kube-api-access-nqftg\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:29.669237 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669211    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-system-cni-dir\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.669237 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669238    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-var-lib-kubelet\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.669376 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669253    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-log-socket\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.669376 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669267    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea45e6d2-1a47-4956-b652-19b428800f42-sys-fs\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:26:29.669376 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669287    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-metrics-certs\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:29.669376 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669311    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/214f4f91-4d6e-46e0-ac1f-d23935cd7215-metrics-certs\") pod \"network-metrics-daemon-zcslg\" (UID: \"214f4f91-4d6e-46e0-ac1f-d23935cd7215\") " pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:26:29.669376 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669335    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-hostroot\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.669376 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669346    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-system-cni-dir\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.669376 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669358    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-slash\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.669376 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669344    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-var-lib-kubelet\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.669376 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669369    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea45e6d2-1a47-4956-b652-19b428800f42-sys-fs\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:26:29.669750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669390    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-log-socket\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.669750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669380    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-run-ovn\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.669750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669406    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-hostroot\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.669750 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:29.669424    5071 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 20:26:29.669750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669433    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-run-ovn\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.669750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669436    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-multus-cni-dir\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.669750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669433    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-slash\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.669750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669463    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-host-run-k8s-cni-cncf-io\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.669750 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:29.669487    5071 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 20:26:29.669750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669491    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-system-cni-dir\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:26:29.669750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669506    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-multus-cni-dir\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.669750 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:29.669518    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-metrics-certs podName:533fb89e-0b69-4c79-9cbf-7baa5f725219 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:30.169500577 +0000 UTC m=+2.086522107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-metrics-certs") pod "router-default-666dd86ff-dnnq5" (UID: "533fb89e-0b69-4c79-9cbf-7baa5f725219") : secret "router-metrics-certs-default" not found
Oct 30 20:26:29.669750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669526    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-system-cni-dir\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:26:29.669750 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:29.669540    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/214f4f91-4d6e-46e0-ac1f-d23935cd7215-metrics-certs podName:214f4f91-4d6e-46e0-ac1f-d23935cd7215 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:30.169534317 +0000 UTC m=+2.086555830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/214f4f91-4d6e-46e0-ac1f-d23935cd7215-metrics-certs") pod "network-metrics-daemon-zcslg" (UID: "214f4f91-4d6e-46e0-ac1f-d23935cd7215") : secret "metrics-daemon-secret" not found
Oct 30 20:26:29.669750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669542    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-host-run-k8s-cni-cncf-io\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.669750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669562    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b87eabe7-2403-4534-987b-ce6ecc6fda94-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tk6nb\" (UID: \"b87eabe7-2403-4534-987b-ce6ecc6fda94\") " pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:29.669750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669588    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b87eabe7-2403-4534-987b-ce6ecc6fda94-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tk6nb\" (UID: \"b87eabe7-2403-4534-987b-ce6ecc6fda94\") " pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:29.669750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669606    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-lib-modules\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.669750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669623    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-cnibin\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:26:29.669750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669637    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-host-run-netns\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.669750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669653    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-multus-conf-dir\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.669750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669677    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-host-var-lib-cni-bin\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.669750 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:29.669684    5071 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:26:29.669750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669700    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea45e6d2-1a47-4956-b652-19b428800f42-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:26:29.669750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669710    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-cnibin\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:26:29.669750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669722    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea45e6d2-1a47-4956-b652-19b428800f42-etc-selinux\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:26:29.669750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669730    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-lib-modules\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.669750 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:29.669740    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b87eabe7-2403-4534-987b-ce6ecc6fda94-insights-runtime-extractor-tls podName:b87eabe7-2403-4534-987b-ce6ecc6fda94 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:30.169724138 +0000 UTC m=+2.086745657 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b87eabe7-2403-4534-987b-ce6ecc6fda94-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tk6nb" (UID: "b87eabe7-2403-4534-987b-ce6ecc6fda94") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:26:29.669750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669730    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-multus-conf-dir\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.669750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669767    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-host-run-netns\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.669750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669773    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea45e6d2-1a47-4956-b652-19b428800f42-etc-selinux\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669787    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-host-var-lib-cni-bin\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669795    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea45e6d2-1a47-4956-b652-19b428800f42-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669806    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-systemd\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669857    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-kubelet\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669869    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-systemd\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669874    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-kubernetes\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669900    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-kubelet\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669914    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-kubernetes\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669914    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-sysctl-conf\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669940    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b87eabe7-2403-4534-987b-ce6ecc6fda94-crio-socket\") pod \"insights-runtime-extractor-tk6nb\" (UID: \"b87eabe7-2403-4534-987b-ce6ecc6fda94\") " pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669957    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/88d9611e-c7ed-490b-b401-530d1649c873-kubelet-config\") pod \"global-pull-secret-syncer-bqh9l\" (UID: \"88d9611e-c7ed-490b-b401-530d1649c873\") " pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.669976    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-registry-tls\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670000    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-host-var-lib-kubelet\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670018    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-sysctl-conf\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670025    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea45e6d2-1a47-4956-b652-19b428800f42-socket-dir\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670049    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-systemd-units\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670058    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-host-var-lib-kubelet\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:29.670065    5071 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:29.670078    5071 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7db8fb6d4-8n6g7: secret "image-registry-tls" not found
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670085    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-systemd-units\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670087    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea45e6d2-1a47-4956-b652-19b428800f42-socket-dir\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670097    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-host-run-multus-certs\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670065    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-host-run-multus-certs\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670019    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/88d9611e-c7ed-490b-b401-530d1649c873-kubelet-config\") pod \"global-pull-secret-syncer-bqh9l\" (UID: \"88d9611e-c7ed-490b-b401-530d1649c873\") " pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:29.670121    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-registry-tls podName:05f22644-0884-434d-b56b-193d9554b505 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:30.17010594 +0000 UTC m=+2.087127490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-registry-tls") pod "image-registry-7db8fb6d4-8n6g7" (UID: "05f22644-0884-434d-b56b-193d9554b505") : secret "image-registry-tls" not found
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670002    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b87eabe7-2403-4534-987b-ce6ecc6fda94-crio-socket\") pod \"insights-runtime-extractor-tk6nb\" (UID: \"b87eabe7-2403-4534-987b-ce6ecc6fda94\") " pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670164    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-modprobe-d\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670168    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b87eabe7-2403-4534-987b-ce6ecc6fda94-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tk6nb\" (UID: \"b87eabe7-2403-4534-987b-ce6ecc6fda94\") " pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670189    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f3f7114f-f896-4fc9-b98a-e47574e7720d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-m9n59\" (UID: \"f3f7114f-f896-4fc9-b98a-e47574e7720d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-m9n59"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670210    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-run-systemd\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670224    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-node-log\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670241    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-cnibin\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670261    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-sysctl-d\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670273    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-modprobe-d\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670275    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-node-log\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670263    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-run-systemd\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:29.670293    5071 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670295    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-cnibin\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:29.670333    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3f7114f-f896-4fc9-b98a-e47574e7720d-samples-operator-tls podName:f3f7114f-f896-4fc9-b98a-e47574e7720d nodeName:}" failed. No retries permitted until 2026-10-30 20:26:30.170320234 +0000 UTC m=+2.087341769 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f3f7114f-f896-4fc9-b98a-e47574e7720d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-m9n59" (UID: "f3f7114f-f896-4fc9-b98a-e47574e7720d") : secret "samples-operator-tls" not found
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670334    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-var-lib-openvswitch\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670284    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-var-lib-openvswitch\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670372    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-run\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670387    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-run-netns\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670397    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-sysctl-d\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670407    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-sys\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670424    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-etc-openvswitch\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670447    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-run-ovn-kubernetes\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670475    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-sys\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670475    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-cni-netd\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670484    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-etc-openvswitch\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670504    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-run\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670508    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-etc-kubernetes\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670507    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-cni-netd\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670507    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-run-ovn-kubernetes\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670480    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-run-netns\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670540    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/111798c9-6959-4dcb-9a8a-db3037f7d662-host-slash\") pod \"iptables-alerter-kwm7g\" (UID: \"111798c9-6959-4dcb-9a8a-db3037f7d662\") " pod="openshift-network-operator/iptables-alerter-kwm7g"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670549    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-etc-kubernetes\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670569    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d5312b70-82df-4e75-a300-b8e1d63a75f3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9rs22\" (UID: \"d5312b70-82df-4e75-a300-b8e1d63a75f3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9rs22"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670588    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-sysconfig\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670615    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/533fb89e-0b69-4c79-9cbf-7baa5f725219-service-ca-bundle\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670589    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/111798c9-6959-4dcb-9a8a-db3037f7d662-host-slash\") pod \"iptables-alerter-kwm7g\" (UID: \"111798c9-6959-4dcb-9a8a-db3037f7d662\") " pod="openshift-network-operator/iptables-alerter-kwm7g"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670652    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-etc-sysconfig\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670665    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-run-openvswitch\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:29.670693    5071 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670703    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/88178646-9467-4225-b362-578f31454922-hosts-file\") pod \"node-resolver-c8422\" (UID: \"88178646-9467-4225-b362-578f31454922\") " pod="openshift-dns/node-resolver-c8422"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:29.670718    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/533fb89e-0b69-4c79-9cbf-7baa5f725219-service-ca-bundle podName:533fb89e-0b69-4c79-9cbf-7baa5f725219 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:30.170705036 +0000 UTC m=+2.087726564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/533fb89e-0b69-4c79-9cbf-7baa5f725219-service-ca-bundle") pod "router-default-666dd86ff-dnnq5" (UID: "533fb89e-0b69-4c79-9cbf-7baa5f725219") : configmap references non-existent config key: service-ca.crt
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:29.670738    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5312b70-82df-4e75-a300-b8e1d63a75f3-cluster-monitoring-operator-tls podName:d5312b70-82df-4e75-a300-b8e1d63a75f3 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:30.170728369 +0000 UTC m=+2.087749883 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d5312b70-82df-4e75-a300-b8e1d63a75f3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9rs22" (UID: "d5312b70-82df-4e75-a300-b8e1d63a75f3") : secret "cluster-monitoring-operator-tls" not found
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670741    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-run-openvswitch\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670744    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/88178646-9467-4225-b362-578f31454922-hosts-file\") pod \"node-resolver-c8422\" (UID: \"88178646-9467-4225-b362-578f31454922\") " pod="openshift-dns/node-resolver-c8422"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670762    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670778    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea45e6d2-1a47-4956-b652-19b428800f42-registration-dir\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670791    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-host\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670804    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-metrics-tls\") pod \"dns-default-h2vh7\" (UID: \"4c7e75b7-c3cd-4791-9d39-9120f6fd95dc\") " pod="openshift-dns/dns-default-h2vh7"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670812    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7187a44e-c9e7-4d2d-be9c-f99a11c1401b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qxwlb\" (UID: \"7187a44e-c9e7-4d2d-be9c-f99a11c1401b\") " pod="openshift-multus/multus-additional-cni-plugins-qxwlb"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670844    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-registry-tls\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670869    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d9d686e0-6edc-4db0-ab0d-4fd855c2e332-host\") pod \"tuned-59qcr\" (UID: \"d9d686e0-6edc-4db0-ab0d-4fd855c2e332\") " pod="openshift-cluster-node-tuning-operator/tuned-59qcr"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:29.670906    5071 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670911    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-multus-socket-dir-parent\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670868    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea45e6d2-1a47-4956-b652-19b428800f42-registration-dir\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670871    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-multus-socket-dir-parent\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:29.670932    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-metrics-tls podName:4c7e75b7-c3cd-4791-9d39-9120f6fd95dc nodeName:}" failed. No retries permitted until 2026-10-30 20:26:30.17092453 +0000 UTC m=+2.087946082 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-metrics-tls") pod "dns-default-h2vh7" (UID: "4c7e75b7-c3cd-4791-9d39-9120f6fd95dc") : secret "dns-default-metrics-tls" not found
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:29.670963    5071 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:29.670971    5071 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-968d68cdf-vwtgc: secret "image-registry-tls" not found
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.670991    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-host-var-lib-cni-multus\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:29.671048    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-registry-tls podName:9f908129-4500-4433-871c-032c29a55dd9 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:30.171032237 +0000 UTC m=+2.088053765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-registry-tls") pod "image-registry-968d68cdf-vwtgc" (UID: "9f908129-4500-4433-871c-032c29a55dd9") : secret "image-registry-tls" not found
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.671092    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d83a8ce1-0cdc-410c-87fb-bfb640dfe702-host-var-lib-cni-multus\") pod \"multus-2b77p\" (UID: \"d83a8ce1-0cdc-410c-87fb-bfb640dfe702\") " pod="openshift-multus/multus-2b77p"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.671094    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b7865f8-cfd2-4938-8621-92b63faee7af-host\") pod \"node-ca-rdqrc\" (UID: \"8b7865f8-cfd2-4938-8621-92b63faee7af\") " pod="openshift-image-registry/node-ca-rdqrc"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.671124    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea45e6d2-1a47-4956-b652-19b428800f42-device-dir\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.671146    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b7865f8-cfd2-4938-8621-92b63faee7af-host\") pod \"node-ca-rdqrc\" (UID: \"8b7865f8-cfd2-4938-8621-92b63faee7af\") " pod="openshift-image-registry/node-ca-rdqrc"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.671149    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.671186    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-cni-bin\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.671117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.671187    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.673707 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.671201    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea45e6d2-1a47-4956-b652-19b428800f42-device-dir\") pod \"aws-ebs-csi-driver-node-xnt95\" (UID: \"ea45e6d2-1a47-4956-b652-19b428800f42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnt95"
Oct 30 20:26:29.673707 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.671210    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/88d9611e-c7ed-490b-b401-530d1649c873-dbus\") pod \"global-pull-secret-syncer-bqh9l\" (UID: \"88d9611e-c7ed-490b-b401-530d1649c873\") " pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:26:29.673707 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.671219    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/80cae79e-b60f-4efe-bb58-e77753a457e5-host-cni-bin\") pod \"ovnkube-node-q4f4h\" (UID: \"80cae79e-b60f-4efe-bb58-e77753a457e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:29.673707 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.671240    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70fa9600-c497-4deb-a133-3cb24f1c8050-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4v8mv\" (UID: \"70fa9600-c497-4deb-a133-3cb24f1c8050\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4v8mv"
Oct 30 20:26:29.673707 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.671288    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/88d9611e-c7ed-490b-b401-530d1649c873-dbus\") pod \"global-pull-secret-syncer-bqh9l\" (UID: \"88d9611e-c7ed-490b-b401-530d1649c873\") " pod="kube-system/global-pull-secret-syncer-bqh9l"
Oct 30 20:26:29.673707 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:29.671349    5071 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 20:26:29.673707 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:29.671399    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70fa9600-c497-4deb-a133-3cb24f1c8050-networking-console-plugin-cert podName:70fa9600-c497-4deb-a133-3cb24f1c8050 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:30.171385962 +0000 UTC m=+2.088407494 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/70fa9600-c497-4deb-a133-3cb24f1c8050-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-4v8mv" (UID: "70fa9600-c497-4deb-a133-3cb24f1c8050") : secret "networking-console-plugin-cert" not found
Oct 30 20:26:29.674578 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.674563    5071 scope.go:122] "RemoveContainer" containerID="b21ea8949c9de8f71605d238975ce27033db9c0ec4dff0ed0d0894a5dbcff357"
Oct 30 20:26:29.779476 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.779449    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:29.944213 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:29.944124    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:30.175669 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:30.175636    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d5312b70-82df-4e75-a300-b8e1d63a75f3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9rs22\" (UID: \"d5312b70-82df-4e75-a300-b8e1d63a75f3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9rs22"
Oct 30 20:26:30.175669 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:30.175670    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/533fb89e-0b69-4c79-9cbf-7baa5f725219-service-ca-bundle\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:30.176254 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:30.175691    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-metrics-tls\") pod \"dns-default-h2vh7\" (UID: \"4c7e75b7-c3cd-4791-9d39-9120f6fd95dc\") " pod="openshift-dns/dns-default-h2vh7"
Oct 30 20:26:30.176254 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:30.175778    5071 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:26:30.176254 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:30.175777    5071 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 20:26:30.176254 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:30.175801    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/533fb89e-0b69-4c79-9cbf-7baa5f725219-service-ca-bundle podName:533fb89e-0b69-4c79-9cbf-7baa5f725219 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:31.175784728 +0000 UTC m=+3.092806259 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/533fb89e-0b69-4c79-9cbf-7baa5f725219-service-ca-bundle") pod "router-default-666dd86ff-dnnq5" (UID: "533fb89e-0b69-4c79-9cbf-7baa5f725219") : configmap references non-existent config key: service-ca.crt
Oct 30 20:26:30.176254 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:30.175848    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-registry-tls\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:30.176254 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:30.175855    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5312b70-82df-4e75-a300-b8e1d63a75f3-cluster-monitoring-operator-tls podName:d5312b70-82df-4e75-a300-b8e1d63a75f3 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:31.175842805 +0000 UTC m=+3.092864320 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d5312b70-82df-4e75-a300-b8e1d63a75f3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9rs22" (UID: "d5312b70-82df-4e75-a300-b8e1d63a75f3") : secret "cluster-monitoring-operator-tls" not found
Oct 30 20:26:30.176254 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:30.175887    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-metrics-tls podName:4c7e75b7-c3cd-4791-9d39-9120f6fd95dc nodeName:}" failed. No retries permitted until 2026-10-30 20:26:31.175864834 +0000 UTC m=+3.092886349 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-metrics-tls") pod "dns-default-h2vh7" (UID: "4c7e75b7-c3cd-4791-9d39-9120f6fd95dc") : secret "dns-default-metrics-tls" not found
Oct 30 20:26:30.176254 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:30.175897    5071 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:26:30.176254 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:30.175909    5071 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-968d68cdf-vwtgc: secret "image-registry-tls" not found
Oct 30 20:26:30.176254 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:30.175945    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-registry-tls podName:9f908129-4500-4433-871c-032c29a55dd9 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:31.175933475 +0000 UTC m=+3.092954990 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-registry-tls") pod "image-registry-968d68cdf-vwtgc" (UID: "9f908129-4500-4433-871c-032c29a55dd9") : secret "image-registry-tls" not found
Oct 30 20:26:30.176254 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:30.175959    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70fa9600-c497-4deb-a133-3cb24f1c8050-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4v8mv\" (UID: \"70fa9600-c497-4deb-a133-3cb24f1c8050\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4v8mv"
Oct 30 20:26:30.176254 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:30.175984    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-metrics-certs\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:30.176254 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:30.176006    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/214f4f91-4d6e-46e0-ac1f-d23935cd7215-metrics-certs\") pod \"network-metrics-daemon-zcslg\" (UID: \"214f4f91-4d6e-46e0-ac1f-d23935cd7215\") " pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:26:30.176254 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:30.176057    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b87eabe7-2403-4534-987b-ce6ecc6fda94-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tk6nb\" (UID: \"b87eabe7-2403-4534-987b-ce6ecc6fda94\") " pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:30.176254 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:30.176060    5071 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 20:26:30.176254 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:30.176123    5071 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:26:30.176254 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:30.176061    5071 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 20:26:30.176254 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:30.176144    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/214f4f91-4d6e-46e0-ac1f-d23935cd7215-metrics-certs podName:214f4f91-4d6e-46e0-ac1f-d23935cd7215 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:31.176133191 +0000 UTC m=+3.093154707 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/214f4f91-4d6e-46e0-ac1f-d23935cd7215-metrics-certs") pod "network-metrics-daemon-zcslg" (UID: "214f4f91-4d6e-46e0-ac1f-d23935cd7215") : secret "metrics-daemon-secret" not found
Oct 30 20:26:30.176254 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:30.176159    5071 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:26:30.176254 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:30.176107    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-registry-tls\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:30.176254 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:30.176171    5071 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7db8fb6d4-8n6g7: secret "image-registry-tls" not found
Oct 30 20:26:30.176254 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:30.176162    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b87eabe7-2403-4534-987b-ce6ecc6fda94-insights-runtime-extractor-tls podName:b87eabe7-2403-4534-987b-ce6ecc6fda94 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:31.17615255 +0000 UTC m=+3.093174073 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b87eabe7-2403-4534-987b-ce6ecc6fda94-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tk6nb" (UID: "b87eabe7-2403-4534-987b-ce6ecc6fda94") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:26:30.176254 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:30.176102    5071 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 20:26:30.176254 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:30.176196    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-metrics-certs podName:533fb89e-0b69-4c79-9cbf-7baa5f725219 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:31.176186924 +0000 UTC m=+3.093208452 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-metrics-certs") pod "router-default-666dd86ff-dnnq5" (UID: "533fb89e-0b69-4c79-9cbf-7baa5f725219") : secret "router-metrics-certs-default" not found
Oct 30 20:26:30.176254 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:30.176212    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-registry-tls podName:05f22644-0884-434d-b56b-193d9554b505 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:31.176205436 +0000 UTC m=+3.093226949 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-registry-tls") pod "image-registry-7db8fb6d4-8n6g7" (UID: "05f22644-0884-434d-b56b-193d9554b505") : secret "image-registry-tls" not found
Oct 30 20:26:30.176254 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:30.176228    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70fa9600-c497-4deb-a133-3cb24f1c8050-networking-console-plugin-cert podName:70fa9600-c497-4deb-a133-3cb24f1c8050 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:31.176221586 +0000 UTC m=+3.093243099 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/70fa9600-c497-4deb-a133-3cb24f1c8050-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-4v8mv" (UID: "70fa9600-c497-4deb-a133-3cb24f1c8050") : secret "networking-console-plugin-cert" not found
Oct 30 20:26:30.176254 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:30.176250    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f3f7114f-f896-4fc9-b98a-e47574e7720d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-m9n59\" (UID: \"f3f7114f-f896-4fc9-b98a-e47574e7720d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-m9n59"
Oct 30 20:26:30.177090 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:30.176314    5071 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 20:26:30.177090 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:30.176335    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3f7114f-f896-4fc9-b98a-e47574e7720d-samples-operator-tls podName:f3f7114f-f896-4fc9-b98a-e47574e7720d nodeName:}" failed. No retries permitted until 2026-10-30 20:26:31.176329133 +0000 UTC m=+3.093350647 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f3f7114f-f896-4fc9-b98a-e47574e7720d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-m9n59" (UID: "f3f7114f-f896-4fc9-b98a-e47574e7720d") : secret "samples-operator-tls" not found
Oct 30 20:26:30.261437 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:30.261368    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8qvb_91fe9e06-0c30-4000-bddb-2ba89cd048cc/console-operator/2.log"
Oct 30 20:26:30.261770 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:30.261753    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8qvb_91fe9e06-0c30-4000-bddb-2ba89cd048cc/console-operator/1.log"
Oct 30 20:26:30.261845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:30.261788    5071 generic.go:356] "Generic (PLEG): container finished" podID="91fe9e06-0c30-4000-bddb-2ba89cd048cc" containerID="74017be9f5939362805c10d5e5fa865895552cfdf93185b30963480a1b686cd7" exitCode=255
Oct 30 20:26:30.261903 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:30.261871    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb" event={"ID":"91fe9e06-0c30-4000-bddb-2ba89cd048cc","Type":"ContainerDied","Data":"74017be9f5939362805c10d5e5fa865895552cfdf93185b30963480a1b686cd7"}
Oct 30 20:26:30.261957 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:30.261940    5071 scope.go:122] "RemoveContainer" containerID="b21ea8949c9de8f71605d238975ce27033db9c0ec4dff0ed0d0894a5dbcff357"
Oct 30 20:26:30.262388 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:30.262369    5071 scope.go:122] "RemoveContainer" containerID="74017be9f5939362805c10d5e5fa865895552cfdf93185b30963480a1b686cd7"
Oct 30 20:26:30.262565 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:30.262542    5071 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-l8qvb_openshift-console-operator(91fe9e06-0c30-4000-bddb-2ba89cd048cc)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb" podUID="91fe9e06-0c30-4000-bddb-2ba89cd048cc"
Oct 30 20:26:30.287422 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:30.287405    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rdqrc_8b7865f8-cfd2-4938-8621-92b63faee7af/node-ca/0.log"
Oct 30 20:26:30.721732 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:30.721706    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:31.004522 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:31.004446    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:31.012610 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:31.012585    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb"
Oct 30 20:26:31.087791 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:31.087761    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mrnvn_c1e8d4f3-e4a1-416c-bbc7-2c76063bfc24/serve-healthcheck-canary/0.log"
Oct 30 20:26:31.185529 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:31.185495    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/533fb89e-0b69-4c79-9cbf-7baa5f725219-service-ca-bundle\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:31.186064 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:31.185540    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-metrics-tls\") pod \"dns-default-h2vh7\" (UID: \"4c7e75b7-c3cd-4791-9d39-9120f6fd95dc\") " pod="openshift-dns/dns-default-h2vh7"
Oct 30 20:26:31.186064 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:31.185560    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-registry-tls\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:31.186064 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:31.185587    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70fa9600-c497-4deb-a133-3cb24f1c8050-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4v8mv\" (UID: \"70fa9600-c497-4deb-a133-3cb24f1c8050\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4v8mv"
Oct 30 20:26:31.186064 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:31.185608    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-metrics-certs\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:31.186064 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:31.185680    5071 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 20:26:31.186064 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:31.185685    5071 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:26:31.186064 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:31.185700    5071 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-968d68cdf-vwtgc: secret "image-registry-tls" not found
Oct 30 20:26:31.186064 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:31.185719    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/214f4f91-4d6e-46e0-ac1f-d23935cd7215-metrics-certs\") pod \"network-metrics-daemon-zcslg\" (UID: \"214f4f91-4d6e-46e0-ac1f-d23935cd7215\") " pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:26:31.186064 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:31.185683    5071 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:26:31.186064 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:31.185760    5071 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 20:26:31.186064 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:31.185728    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70fa9600-c497-4deb-a133-3cb24f1c8050-networking-console-plugin-cert podName:70fa9600-c497-4deb-a133-3cb24f1c8050 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:33.185716403 +0000 UTC m=+5.102737917 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/70fa9600-c497-4deb-a133-3cb24f1c8050-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-4v8mv" (UID: "70fa9600-c497-4deb-a133-3cb24f1c8050") : secret "networking-console-plugin-cert" not found
Oct 30 20:26:31.186064 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:31.185800    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-metrics-certs podName:533fb89e-0b69-4c79-9cbf-7baa5f725219 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:33.185785744 +0000 UTC m=+5.102807259 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-metrics-certs") pod "router-default-666dd86ff-dnnq5" (UID: "533fb89e-0b69-4c79-9cbf-7baa5f725219") : secret "router-metrics-certs-default" not found
Oct 30 20:26:31.186064 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:31.185809    5071 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 20:26:31.186064 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:31.185845    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-registry-tls podName:9f908129-4500-4433-871c-032c29a55dd9 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:33.185814567 +0000 UTC m=+5.102836097 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-registry-tls") pod "image-registry-968d68cdf-vwtgc" (UID: "9f908129-4500-4433-871c-032c29a55dd9") : secret "image-registry-tls" not found
Oct 30 20:26:31.186064 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:31.185877    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-metrics-tls podName:4c7e75b7-c3cd-4791-9d39-9120f6fd95dc nodeName:}" failed. No retries permitted until 2026-10-30 20:26:33.185861557 +0000 UTC m=+5.102883082 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-metrics-tls") pod "dns-default-h2vh7" (UID: "4c7e75b7-c3cd-4791-9d39-9120f6fd95dc") : secret "dns-default-metrics-tls" not found
Oct 30 20:26:31.186064 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:31.185901    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/214f4f91-4d6e-46e0-ac1f-d23935cd7215-metrics-certs podName:214f4f91-4d6e-46e0-ac1f-d23935cd7215 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:33.185892186 +0000 UTC m=+5.102913704 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/214f4f91-4d6e-46e0-ac1f-d23935cd7215-metrics-certs") pod "network-metrics-daemon-zcslg" (UID: "214f4f91-4d6e-46e0-ac1f-d23935cd7215") : secret "metrics-daemon-secret" not found
Oct 30 20:26:31.186064 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:31.185958    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b87eabe7-2403-4534-987b-ce6ecc6fda94-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tk6nb\" (UID: \"b87eabe7-2403-4534-987b-ce6ecc6fda94\") " pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:31.186064 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:31.186012    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-registry-tls\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:31.186064 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:31.186037    5071 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:26:31.186064 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:31.186070    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b87eabe7-2403-4534-987b-ce6ecc6fda94-insights-runtime-extractor-tls podName:b87eabe7-2403-4534-987b-ce6ecc6fda94 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:33.186061688 +0000 UTC m=+5.103083202 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b87eabe7-2403-4534-987b-ce6ecc6fda94-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tk6nb" (UID: "b87eabe7-2403-4534-987b-ce6ecc6fda94") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:26:31.186064 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:31.186040    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f3f7114f-f896-4fc9-b98a-e47574e7720d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-m9n59\" (UID: \"f3f7114f-f896-4fc9-b98a-e47574e7720d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-m9n59"
Oct 30 20:26:31.186804 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:31.186091    5071 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 20:26:31.186804 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:31.186131    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3f7114f-f896-4fc9-b98a-e47574e7720d-samples-operator-tls podName:f3f7114f-f896-4fc9-b98a-e47574e7720d nodeName:}" failed. No retries permitted until 2026-10-30 20:26:33.186120689 +0000 UTC m=+5.103142217 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f3f7114f-f896-4fc9-b98a-e47574e7720d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-m9n59" (UID: "f3f7114f-f896-4fc9-b98a-e47574e7720d") : secret "samples-operator-tls" not found
Oct 30 20:26:31.186804 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:31.186130    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d5312b70-82df-4e75-a300-b8e1d63a75f3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9rs22\" (UID: \"d5312b70-82df-4e75-a300-b8e1d63a75f3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9rs22"
Oct 30 20:26:31.186804 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:31.186136    5071 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:26:31.186804 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:31.186178    5071 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7db8fb6d4-8n6g7: secret "image-registry-tls" not found
Oct 30 20:26:31.186804 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:31.186177    5071 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 20:26:31.186804 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:31.186201    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-registry-tls podName:05f22644-0884-434d-b56b-193d9554b505 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:33.186193042 +0000 UTC m=+5.103214602 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-registry-tls") pod "image-registry-7db8fb6d4-8n6g7" (UID: "05f22644-0884-434d-b56b-193d9554b505") : secret "image-registry-tls" not found
Oct 30 20:26:31.186804 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:31.186213    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5312b70-82df-4e75-a300-b8e1d63a75f3-cluster-monitoring-operator-tls podName:d5312b70-82df-4e75-a300-b8e1d63a75f3 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:33.186206761 +0000 UTC m=+5.103228320 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d5312b70-82df-4e75-a300-b8e1d63a75f3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9rs22" (UID: "d5312b70-82df-4e75-a300-b8e1d63a75f3") : secret "cluster-monitoring-operator-tls" not found
Oct 30 20:26:31.186804 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:31.186270    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/533fb89e-0b69-4c79-9cbf-7baa5f725219-service-ca-bundle\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:31.266598 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:31.266524    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8qvb_91fe9e06-0c30-4000-bddb-2ba89cd048cc/console-operator/2.log"
Oct 30 20:26:31.266882 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:31.266868    5071 scope.go:122] "RemoveContainer" containerID="74017be9f5939362805c10d5e5fa865895552cfdf93185b30963480a1b686cd7"
Oct 30 20:26:31.267057 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:31.267041    5071 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-l8qvb_openshift-console-operator(91fe9e06-0c30-4000-bddb-2ba89cd048cc)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb" podUID="91fe9e06-0c30-4000-bddb-2ba89cd048cc"
Oct 30 20:26:31.286073 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:31.286045    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:31.294781 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:31.294760    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-hbm7v_50f563bf-3336-49c6-9708-041c41b41c53/migrator/0.log"
Oct 30 20:26:31.472347 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:31.472312    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:31.487575 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:31.487549    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-lfrbc_fe578b91-829c-450a-a60a-0f4e4b6e8c63/migrator/0.log"
Oct 30 20:26:31.633701 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:31.633661    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:31.691812 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:31.691785    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf_ce6c7b90-6b06-4c81-bf95-539b71e2aaed/kube-storage-version-migrator-operator/0.log"
Oct 30 20:26:31.799977 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:31.799949    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:31.970802 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:31.970722    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:32.997750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:32.997716    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:33.205705 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:33.205672    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d5312b70-82df-4e75-a300-b8e1d63a75f3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9rs22\" (UID: \"d5312b70-82df-4e75-a300-b8e1d63a75f3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9rs22"
Oct 30 20:26:33.205936 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:33.205728    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-metrics-tls\") pod \"dns-default-h2vh7\" (UID: \"4c7e75b7-c3cd-4791-9d39-9120f6fd95dc\") " pod="openshift-dns/dns-default-h2vh7"
Oct 30 20:26:33.205936 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:33.205755    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-registry-tls\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:33.205936 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:33.205797    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70fa9600-c497-4deb-a133-3cb24f1c8050-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4v8mv\" (UID: \"70fa9600-c497-4deb-a133-3cb24f1c8050\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4v8mv"
Oct 30 20:26:33.205936 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:33.205851    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-metrics-certs\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:33.205936 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:33.205859    5071 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 20:26:33.205936 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:33.205874    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/214f4f91-4d6e-46e0-ac1f-d23935cd7215-metrics-certs\") pod \"network-metrics-daemon-zcslg\" (UID: \"214f4f91-4d6e-46e0-ac1f-d23935cd7215\") " pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:26:33.205936 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:33.205889    5071 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:26:33.205936 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:33.205915    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b87eabe7-2403-4534-987b-ce6ecc6fda94-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tk6nb\" (UID: \"b87eabe7-2403-4534-987b-ce6ecc6fda94\") " pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:33.205936 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:33.205933    5071 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:26:33.206284 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:33.205955    5071 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-968d68cdf-vwtgc: secret "image-registry-tls" not found
Oct 30 20:26:33.206284 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:33.205968    5071 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 20:26:33.206284 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:33.205968    5071 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 20:26:33.206284 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:33.205988    5071 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:26:33.206284 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:33.205939    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5312b70-82df-4e75-a300-b8e1d63a75f3-cluster-monitoring-operator-tls podName:d5312b70-82df-4e75-a300-b8e1d63a75f3 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:37.205919496 +0000 UTC m=+9.122941028 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d5312b70-82df-4e75-a300-b8e1d63a75f3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9rs22" (UID: "d5312b70-82df-4e75-a300-b8e1d63a75f3") : secret "cluster-monitoring-operator-tls" not found
Oct 30 20:26:33.206284 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:33.206035    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-metrics-tls podName:4c7e75b7-c3cd-4791-9d39-9120f6fd95dc nodeName:}" failed. No retries permitted until 2026-10-30 20:26:37.206018742 +0000 UTC m=+9.123040261 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-metrics-tls") pod "dns-default-h2vh7" (UID: "4c7e75b7-c3cd-4791-9d39-9120f6fd95dc") : secret "dns-default-metrics-tls" not found
Oct 30 20:26:33.206284 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:33.206053    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-registry-tls podName:9f908129-4500-4433-871c-032c29a55dd9 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:37.206044553 +0000 UTC m=+9.123066068 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-registry-tls") pod "image-registry-968d68cdf-vwtgc" (UID: "9f908129-4500-4433-871c-032c29a55dd9") : secret "image-registry-tls" not found
Oct 30 20:26:33.206284 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:33.206070    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70fa9600-c497-4deb-a133-3cb24f1c8050-networking-console-plugin-cert podName:70fa9600-c497-4deb-a133-3cb24f1c8050 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:37.206061755 +0000 UTC m=+9.123083281 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/70fa9600-c497-4deb-a133-3cb24f1c8050-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-4v8mv" (UID: "70fa9600-c497-4deb-a133-3cb24f1c8050") : secret "networking-console-plugin-cert" not found
Oct 30 20:26:33.206284 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:33.206086    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-metrics-certs podName:533fb89e-0b69-4c79-9cbf-7baa5f725219 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:37.206079423 +0000 UTC m=+9.123100952 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-metrics-certs") pod "router-default-666dd86ff-dnnq5" (UID: "533fb89e-0b69-4c79-9cbf-7baa5f725219") : secret "router-metrics-certs-default" not found
Oct 30 20:26:33.206284 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:33.206102    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b87eabe7-2403-4534-987b-ce6ecc6fda94-insights-runtime-extractor-tls podName:b87eabe7-2403-4534-987b-ce6ecc6fda94 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:37.206094927 +0000 UTC m=+9.123116441 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b87eabe7-2403-4534-987b-ce6ecc6fda94-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tk6nb" (UID: "b87eabe7-2403-4534-987b-ce6ecc6fda94") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:26:33.206284 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:33.206162    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-registry-tls\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:33.206284 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:33.206191    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f3f7114f-f896-4fc9-b98a-e47574e7720d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-m9n59\" (UID: \"f3f7114f-f896-4fc9-b98a-e47574e7720d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-m9n59"
Oct 30 20:26:33.206284 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:33.206257    5071 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 20:26:33.206284 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:33.206276    5071 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 20:26:33.206284 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:33.206288    5071 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7db8fb6d4-8n6g7: secret "image-registry-tls" not found
Oct 30 20:26:33.206284 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:33.206287    5071 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 20:26:33.206782 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:33.206323    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-registry-tls podName:05f22644-0884-434d-b56b-193d9554b505 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:37.206313722 +0000 UTC m=+9.123335237 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-registry-tls") pod "image-registry-7db8fb6d4-8n6g7" (UID: "05f22644-0884-434d-b56b-193d9554b505") : secret "image-registry-tls" not found
Oct 30 20:26:33.206782 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:33.206337    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3f7114f-f896-4fc9-b98a-e47574e7720d-samples-operator-tls podName:f3f7114f-f896-4fc9-b98a-e47574e7720d nodeName:}" failed. No retries permitted until 2026-10-30 20:26:37.206329816 +0000 UTC m=+9.123351330 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f3f7114f-f896-4fc9-b98a-e47574e7720d-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-m9n59" (UID: "f3f7114f-f896-4fc9-b98a-e47574e7720d") : secret "samples-operator-tls" not found
Oct 30 20:26:33.208430 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:33.208412    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/214f4f91-4d6e-46e0-ac1f-d23935cd7215-metrics-certs\") pod \"network-metrics-daemon-zcslg\" (UID: \"214f4f91-4d6e-46e0-ac1f-d23935cd7215\") " pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:26:33.222271 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:33.222246    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:33.281614 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:33.281554    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zcslg"
Oct 30 20:26:33.410396 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:33.410364    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:33.599010 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:33.598972    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:33.648754 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:33.648648    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zcslg"]
Oct 30 20:26:33.651440 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:33.651410    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod214f4f91_4d6e_46e0_ac1f_d23935cd7215.slice/crio-401ae4c1924e1c84875b55484618bd83925503fb4302e64c23b8839cb635c5ee WatchSource:0}: Error finding container 401ae4c1924e1c84875b55484618bd83925503fb4302e64c23b8839cb635c5ee: Status 404 returned error can't find the container with id 401ae4c1924e1c84875b55484618bd83925503fb4302e64c23b8839cb635c5ee
Oct 30 20:26:33.653393 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:33.653378    5071 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:26:33.794064 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:33.794026    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:34.278000 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:34.277923    5071 generic.go:356] "Generic (PLEG): container finished" podID="ce6c7b90-6b06-4c81-bf95-539b71e2aaed" containerID="b8586be1520b81fb122e2d0c3f13e8c1fa8fefef1764be88b7ff904c090e57f6" exitCode=0
Oct 30 20:26:34.278395 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:34.277997    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf" event={"ID":"ce6c7b90-6b06-4c81-bf95-539b71e2aaed","Type":"ContainerDied","Data":"b8586be1520b81fb122e2d0c3f13e8c1fa8fefef1764be88b7ff904c090e57f6"}
Oct 30 20:26:34.278395 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:34.278337    5071 scope.go:122] "RemoveContainer" containerID="b8586be1520b81fb122e2d0c3f13e8c1fa8fefef1764be88b7ff904c090e57f6"
Oct 30 20:26:34.279284 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:34.279260    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zcslg" event={"ID":"214f4f91-4d6e-46e0-ac1f-d23935cd7215","Type":"ContainerStarted","Data":"401ae4c1924e1c84875b55484618bd83925503fb4302e64c23b8839cb635c5ee"}
Oct 30 20:26:34.645642 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:34.645603    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:34.806135 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:34.806089    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:34.974983 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:34.974897    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:35.197961 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:35.197933    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:35.288175 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:35.288147    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zcslg" event={"ID":"214f4f91-4d6e-46e0-ac1f-d23935cd7215","Type":"ContainerStarted","Data":"c19ea97ca709a5ed6fddf97d819d5ce1bec5d34f7e0161545f4d57af102f5e2d"}
Oct 30 20:26:35.289967 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:35.289946    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tcrkf" event={"ID":"ce6c7b90-6b06-4c81-bf95-539b71e2aaed","Type":"ContainerStarted","Data":"3278ea489b0e04cadf35c384d828065d0213cbc58884cea3bb9e8a89a2b83b95"}
Oct 30 20:26:35.375327 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:35.375293    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:35.544088 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:35.544052    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:35.685464 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:35.685433    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:35.685658 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:35.685645    5071 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 20:26:35.704242 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:35.704216    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:35.878866 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:35.878757    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:36.294909 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:36.294876    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zcslg" event={"ID":"214f4f91-4d6e-46e0-ac1f-d23935cd7215","Type":"ContainerStarted","Data":"1d859bfc86dd5183efe54d74d1c84e589f6148c9f3044300cd17d6cff0c2abb2"}
Oct 30 20:26:37.002781 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:37.002747    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:37.211571 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:37.211541    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:37.214722 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:37.214695    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q4f4h"
Oct 30 20:26:37.244124 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:37.244090    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-registry-tls\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:37.244271 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:37.244137    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f3f7114f-f896-4fc9-b98a-e47574e7720d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-m9n59\" (UID: \"f3f7114f-f896-4fc9-b98a-e47574e7720d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-m9n59"
Oct 30 20:26:37.244271 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:37.244211    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d5312b70-82df-4e75-a300-b8e1d63a75f3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9rs22\" (UID: \"d5312b70-82df-4e75-a300-b8e1d63a75f3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9rs22"
Oct 30 20:26:37.244438 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:37.244280    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-metrics-tls\") pod \"dns-default-h2vh7\" (UID: \"4c7e75b7-c3cd-4791-9d39-9120f6fd95dc\") " pod="openshift-dns/dns-default-h2vh7"
Oct 30 20:26:37.244438 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:37.244311    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-registry-tls\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:37.244438 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:37.244359    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70fa9600-c497-4deb-a133-3cb24f1c8050-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4v8mv\" (UID: \"70fa9600-c497-4deb-a133-3cb24f1c8050\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4v8mv"
Oct 30 20:26:37.244438 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:37.244368    5071 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 20:26:37.244438 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:37.244392    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-metrics-certs\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:37.244680 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:37.244449    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5312b70-82df-4e75-a300-b8e1d63a75f3-cluster-monitoring-operator-tls podName:d5312b70-82df-4e75-a300-b8e1d63a75f3 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:45.244427413 +0000 UTC m=+17.161448947 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d5312b70-82df-4e75-a300-b8e1d63a75f3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-9rs22" (UID: "d5312b70-82df-4e75-a300-b8e1d63a75f3") : secret "cluster-monitoring-operator-tls" not found
Oct 30 20:26:37.244680 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:37.244507    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b87eabe7-2403-4534-987b-ce6ecc6fda94-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tk6nb\" (UID: \"b87eabe7-2403-4534-987b-ce6ecc6fda94\") " pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:37.244680 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:37.244624    5071 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:26:37.244680 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:37.244658    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b87eabe7-2403-4534-987b-ce6ecc6fda94-insights-runtime-extractor-tls podName:b87eabe7-2403-4534-987b-ce6ecc6fda94 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:45.244647584 +0000 UTC m=+17.161669117 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b87eabe7-2403-4534-987b-ce6ecc6fda94-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tk6nb" (UID: "b87eabe7-2403-4534-987b-ce6ecc6fda94") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:26:37.245054 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:37.245032    5071 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 20:26:37.245136 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:37.245086    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-metrics-certs podName:533fb89e-0b69-4c79-9cbf-7baa5f725219 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:45.245071965 +0000 UTC m=+17.162093496 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-metrics-certs") pod "router-default-666dd86ff-dnnq5" (UID: "533fb89e-0b69-4c79-9cbf-7baa5f725219") : secret "router-metrics-certs-default" not found
Oct 30 20:26:37.247343 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:37.247318    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-registry-tls\") pod \"image-registry-7db8fb6d4-8n6g7\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") " pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:37.247523 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:37.247501    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c7e75b7-c3cd-4791-9d39-9120f6fd95dc-metrics-tls\") pod \"dns-default-h2vh7\" (UID: \"4c7e75b7-c3cd-4791-9d39-9120f6fd95dc\") " pod="openshift-dns/dns-default-h2vh7"
Oct 30 20:26:37.247690 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:37.247668    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f3f7114f-f896-4fc9-b98a-e47574e7720d-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-m9n59\" (UID: \"f3f7114f-f896-4fc9-b98a-e47574e7720d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-m9n59"
Oct 30 20:26:37.247894 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:37.247876    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70fa9600-c497-4deb-a133-3cb24f1c8050-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4v8mv\" (UID: \"70fa9600-c497-4deb-a133-3cb24f1c8050\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4v8mv"
Oct 30 20:26:37.248013 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:37.247994    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-registry-tls\") pod \"image-registry-968d68cdf-vwtgc\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") " pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:37.381813 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:37.381783    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pfvnn"
Oct 30 20:26:37.401700 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:37.401670    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:37.474369 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:37.474342    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-m9n59"
Oct 30 20:26:37.474508 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:37.474495    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:37.476513 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:37.476489    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-4v8mv"
Oct 30 20:26:37.485718 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:37.485703    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:37.486800 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:37.486777    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h2vh7"
Oct 30 20:26:37.598686 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:37.598623    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:37.801613 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:37.801569    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:37.915261 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:37.915230    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-m9n59"]
Oct 30 20:26:37.964908 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:37.964887    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-968d68cdf-vwtgc"]
Oct 30 20:26:37.967143 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:37.967124    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h2vh7"]
Oct 30 20:26:37.967508 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:37.967477    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f908129_4500_4433_871c_032c29a55dd9.slice/crio-a264ea4a1ac8f9f1b74cc603b8b44e7b059fb275cf1c1fc56bfdc5efca7b7a7a WatchSource:0}: Error finding container a264ea4a1ac8f9f1b74cc603b8b44e7b059fb275cf1c1fc56bfdc5efca7b7a7a: Status 404 returned error can't find the container with id a264ea4a1ac8f9f1b74cc603b8b44e7b059fb275cf1c1fc56bfdc5efca7b7a7a
Oct 30 20:26:37.970651 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:37.970351    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7db8fb6d4-8n6g7"]
Oct 30 20:26:37.972770 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:37.972749    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f22644_0884_434d_b56b_193d9554b505.slice/crio-5e72fbd9e0a01d8fc01f3e5098fc29f32902db1d36ca6609e1599f8daa23a78c WatchSource:0}: Error finding container 5e72fbd9e0a01d8fc01f3e5098fc29f32902db1d36ca6609e1599f8daa23a78c: Status 404 returned error can't find the container with id 5e72fbd9e0a01d8fc01f3e5098fc29f32902db1d36ca6609e1599f8daa23a78c
Oct 30 20:26:38.010648 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:38.010622    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-4v8mv"]
Oct 30 20:26:38.014430 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:38.014404    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70fa9600_c497_4deb_a133_3cb24f1c8050.slice/crio-34314658af54556351d0ca3c3eb1dc4805983c065a6f26acf87a11f28d525d6a WatchSource:0}: Error finding container 34314658af54556351d0ca3c3eb1dc4805983c065a6f26acf87a11f28d525d6a: Status 404 returned error can't find the container with id 34314658af54556351d0ca3c3eb1dc4805983c065a6f26acf87a11f28d525d6a
Oct 30 20:26:38.162178 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:38.162091    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:38.308386 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:38.308348    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-968d68cdf-vwtgc" event={"ID":"9f908129-4500-4433-871c-032c29a55dd9","Type":"ContainerStarted","Data":"5c38e1613503f4bf4f84774a2192eda53162f86f31dd7d5aa81641c2e532a377"}
Oct 30 20:26:38.308566 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:38.308406    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-968d68cdf-vwtgc" event={"ID":"9f908129-4500-4433-871c-032c29a55dd9","Type":"ContainerStarted","Data":"a264ea4a1ac8f9f1b74cc603b8b44e7b059fb275cf1c1fc56bfdc5efca7b7a7a"}
Oct 30 20:26:38.310490 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:38.310440    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-m9n59" event={"ID":"f3f7114f-f896-4fc9-b98a-e47574e7720d","Type":"ContainerStarted","Data":"635bbf1ca67037f627ab02d4e87439136c8c0c4883eaf44ead9b7190e7afe8d9"}
Oct 30 20:26:38.315354 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:38.315325    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h2vh7" event={"ID":"4c7e75b7-c3cd-4791-9d39-9120f6fd95dc","Type":"ContainerStarted","Data":"230f95c2ea63d594488a3759c7bac86c28e2f309a2d9b24521ce54d5a2bcd067"}
Oct 30 20:26:38.321104 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:38.321080    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-4v8mv" event={"ID":"70fa9600-c497-4deb-a133-3cb24f1c8050","Type":"ContainerStarted","Data":"34314658af54556351d0ca3c3eb1dc4805983c065a6f26acf87a11f28d525d6a"}
Oct 30 20:26:38.328003 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:38.327977    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7" event={"ID":"05f22644-0884-434d-b56b-193d9554b505","Type":"ContainerStarted","Data":"14edc385f1899eecce2b388d841c8ca7dcb619a255508477ab4968ec0efd1811"}
Oct 30 20:26:38.328105 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:38.328016    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7" event={"ID":"05f22644-0884-434d-b56b-193d9554b505","Type":"ContainerStarted","Data":"5e72fbd9e0a01d8fc01f3e5098fc29f32902db1d36ca6609e1599f8daa23a78c"}
Oct 30 20:26:38.445987 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:38.445900    5071 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb"
Oct 30 20:26:38.446341 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:38.446257    5071 scope.go:122] "RemoveContainer" containerID="74017be9f5939362805c10d5e5fa865895552cfdf93185b30963480a1b686cd7"
Oct 30 20:26:38.446448 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:26:38.446429    5071 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-l8qvb_openshift-console-operator(91fe9e06-0c30-4000-bddb-2ba89cd048cc)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb" podUID="91fe9e06-0c30-4000-bddb-2ba89cd048cc"
Oct 30 20:26:39.019437 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:39.019394    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:39.240492 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:39.240455    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:39.332855 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:39.332811    5071 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 20:26:39.333456 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:39.333435    5071 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 20:26:39.406179 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:39.406084    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:39.566847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:39.566785    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:39.734872 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:39.734847    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:39.897576 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:39.897533    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:40.194779 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:40.194452    5071 scope.go:122] "RemoveContainer" containerID="74017be9f5939362805c10d5e5fa865895552cfdf93185b30963480a1b686cd7"
Oct 30 20:26:40.338656 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:40.338565    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-4v8mv" event={"ID":"70fa9600-c497-4deb-a133-3cb24f1c8050","Type":"ContainerStarted","Data":"94899478c9d1adecfec4ee147384b77e5ff0f48b7d6312c74717288f81746f49"}
Oct 30 20:26:40.340880 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:40.340851    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8qvb_91fe9e06-0c30-4000-bddb-2ba89cd048cc/console-operator/2.log"
Oct 30 20:26:40.341016 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:40.340983    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb" event={"ID":"91fe9e06-0c30-4000-bddb-2ba89cd048cc","Type":"ContainerStarted","Data":"cc3e3950006aae738b712236105651e553ccd03023cd4b02b3e7b22459eb5d57"}
Oct 30 20:26:40.341321 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:40.341297    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb"
Oct 30 20:26:40.342656 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:40.342627    5071 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-l8qvb container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.14:8443/readyz\": dial tcp 10.132.0.14:8443: connect: connection refused" start-of-body=
Oct 30 20:26:40.342768 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:40.342672    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb" podUID="91fe9e06-0c30-4000-bddb-2ba89cd048cc" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.14:8443/readyz\": dial tcp 10.132.0.14:8443: connect: connection refused"
Oct 30 20:26:40.343099 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:40.343080    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-m9n59" event={"ID":"f3f7114f-f896-4fc9-b98a-e47574e7720d","Type":"ContainerStarted","Data":"4ba0bf5479ffef480188f3b0cc8599e307d39eba6823ac527a23f1502b90396c"}
Oct 30 20:26:40.343196 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:40.343121    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-m9n59" event={"ID":"f3f7114f-f896-4fc9-b98a-e47574e7720d","Type":"ContainerStarted","Data":"e09f3db5cf8b2d806478297ccb441894bc61d99a619fdeb1390ee20ac86db986"}
Oct 30 20:26:41.349036 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:41.348983    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h2vh7" event={"ID":"4c7e75b7-c3cd-4791-9d39-9120f6fd95dc","Type":"ContainerStarted","Data":"c6aef674be63113b8e2b9c7221cac3877799e0a8b775eb5a37a0cd8740089acf"}
Oct 30 20:26:41.349498 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:41.349065    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h2vh7" event={"ID":"4c7e75b7-c3cd-4791-9d39-9120f6fd95dc","Type":"ContainerStarted","Data":"5c4b23b36a68870962568cab6f817d741e0633b21a3201b121b997f0ed5ce30b"}
Oct 30 20:26:41.349498 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:41.349110    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-h2vh7"
Oct 30 20:26:41.353412 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:41.353394    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-l8qvb"
Oct 30 20:26:41.634239 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:41.634162    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:41.861344 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:41.861308    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:42.042554 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:42.042498    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:42.278462 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:42.278428    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:42.464005 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:42.463907    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:42.754626 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:42.754530    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:42.967884 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:42.967847    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:43.522737 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:43.522706    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:26:44.015363 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:44.015332    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4khkm"
Oct 30 20:26:44.029774 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:44.029750    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:26:45.316086 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:45.316047    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-metrics-certs\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:45.316573 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:45.316096    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b87eabe7-2403-4534-987b-ce6ecc6fda94-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tk6nb\" (UID: \"b87eabe7-2403-4534-987b-ce6ecc6fda94\") " pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:45.316573 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:45.316140    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d5312b70-82df-4e75-a300-b8e1d63a75f3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9rs22\" (UID: \"d5312b70-82df-4e75-a300-b8e1d63a75f3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9rs22"
Oct 30 20:26:45.318637 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:45.318598    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d5312b70-82df-4e75-a300-b8e1d63a75f3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-9rs22\" (UID: \"d5312b70-82df-4e75-a300-b8e1d63a75f3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9rs22"
Oct 30 20:26:45.318770 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:45.318689    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b87eabe7-2403-4534-987b-ce6ecc6fda94-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tk6nb\" (UID: \"b87eabe7-2403-4534-987b-ce6ecc6fda94\") " pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:45.318770 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:45.318689    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533fb89e-0b69-4c79-9cbf-7baa5f725219-metrics-certs\") pod \"router-default-666dd86ff-dnnq5\" (UID: \"533fb89e-0b69-4c79-9cbf-7baa5f725219\") " pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:45.375773 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:45.375745    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:45.573751 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:45.573671    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:45.573925 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:45.573751    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9rs22"
Oct 30 20:26:45.589388 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:45.589364    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tk6nb"
Oct 30 20:26:46.007899 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:46.007879    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-9rs22"]
Oct 30 20:26:46.010246 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:46.010221    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5312b70_82df_4e75_a300_b8e1d63a75f3.slice/crio-3339d05bad0d10f77308e136a8ece359b7430bce8702720d80ac5c006f023f56 WatchSource:0}: Error finding container 3339d05bad0d10f77308e136a8ece359b7430bce8702720d80ac5c006f023f56: Status 404 returned error can't find the container with id 3339d05bad0d10f77308e136a8ece359b7430bce8702720d80ac5c006f023f56
Oct 30 20:26:46.049754 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:46.049729    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod533fb89e_0b69_4c79_9cbf_7baa5f725219.slice/crio-b08bd50599b383a0ac8cca630c7bf9a887d9227d78cfa7354cb30df3333bb23e WatchSource:0}: Error finding container b08bd50599b383a0ac8cca630c7bf9a887d9227d78cfa7354cb30df3333bb23e: Status 404 returned error can't find the container with id b08bd50599b383a0ac8cca630c7bf9a887d9227d78cfa7354cb30df3333bb23e
Oct 30 20:26:46.057557 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:46.057529    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-666dd86ff-dnnq5"]
Oct 30 20:26:46.058490 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:46.058472    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tk6nb"]
Oct 30 20:26:46.061844 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:46.061807    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb87eabe7_2403_4534_987b_ce6ecc6fda94.slice/crio-70ad3f2276d0d81bcefa1f2a32231a05125abdf60323c8d55ce0e2f7aec405db WatchSource:0}: Error finding container 70ad3f2276d0d81bcefa1f2a32231a05125abdf60323c8d55ce0e2f7aec405db: Status 404 returned error can't find the container with id 70ad3f2276d0d81bcefa1f2a32231a05125abdf60323c8d55ce0e2f7aec405db
Oct 30 20:26:46.371133 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:46.371096    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9rs22" event={"ID":"d5312b70-82df-4e75-a300-b8e1d63a75f3","Type":"ContainerStarted","Data":"3339d05bad0d10f77308e136a8ece359b7430bce8702720d80ac5c006f023f56"}
Oct 30 20:26:46.372658 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:46.372636    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tk6nb" event={"ID":"b87eabe7-2403-4534-987b-ce6ecc6fda94","Type":"ContainerStarted","Data":"ab75522df0a4db29da62954c90e2c97c43cad147422ed85a8a725d65135db566"}
Oct 30 20:26:46.372776 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:46.372683    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tk6nb" event={"ID":"b87eabe7-2403-4534-987b-ce6ecc6fda94","Type":"ContainerStarted","Data":"70ad3f2276d0d81bcefa1f2a32231a05125abdf60323c8d55ce0e2f7aec405db"}
Oct 30 20:26:46.374258 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:46.374228    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-666dd86ff-dnnq5" event={"ID":"533fb89e-0b69-4c79-9cbf-7baa5f725219","Type":"ContainerStarted","Data":"46219ddffd710d98ccb25f890f027ef46cf6b4dac7c92aa3b45e5d8e92ee93fa"}
Oct 30 20:26:46.374365 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:46.374262    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-666dd86ff-dnnq5" event={"ID":"533fb89e-0b69-4c79-9cbf-7baa5f725219","Type":"ContainerStarted","Data":"b08bd50599b383a0ac8cca630c7bf9a887d9227d78cfa7354cb30df3333bb23e"}
Oct 30 20:26:46.574478 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:46.574442    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:46.576980 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:46.576945    5071 patch_prober.go:29] interesting pod/router-default-666dd86ff-dnnq5 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld
Oct 30 20:26:46.576980 ip-10-0-139-149 kubenswrapper[5071]: [+]has-synced ok
Oct 30 20:26:46.576980 ip-10-0-139-149 kubenswrapper[5071]: [+]process-running ok
Oct 30 20:26:46.576980 ip-10-0-139-149 kubenswrapper[5071]: healthz check failed
Oct 30 20:26:46.577189 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:46.576998    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-666dd86ff-dnnq5" podUID="533fb89e-0b69-4c79-9cbf-7baa5f725219" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Oct 30 20:26:47.009597 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.009556    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-x6wrc"]
Oct 30 20:26:47.019608 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.019580    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-x6wrc"
Oct 30 20:26:47.029032 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.029004    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-968d68cdf-vwtgc"]
Oct 30 20:26:47.079020 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.078986    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 20:26:47.079020 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.078986    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 20:26:47.081739 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.081629    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-trtdm\""
Oct 30 20:26:47.094581 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.094551    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:47.110690 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.110659    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-86456d9bbc-gxnhh"]
Oct 30 20:26:47.114104 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.114083    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86456d9bbc-gxnhh"
Oct 30 20:26:47.131252 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.131224    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-x6wrc"]
Oct 30 20:26:47.132106 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.131545    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02ea9da9-c742-4322-9f01-f0f7b8f184ef-tmp\") pod \"downloads-6dd789c8b7-x6wrc\" (UID: \"02ea9da9-c742-4322-9f01-f0f7b8f184ef\") " pod="openshift-console/downloads-6dd789c8b7-x6wrc"
Oct 30 20:26:47.132106 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.131574    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8skh\" (UniqueName: \"kubernetes.io/projected/02ea9da9-c742-4322-9f01-f0f7b8f184ef-kube-api-access-m8skh\") pod \"downloads-6dd789c8b7-x6wrc\" (UID: \"02ea9da9-c742-4322-9f01-f0f7b8f184ef\") " pod="openshift-console/downloads-6dd789c8b7-x6wrc"
Oct 30 20:26:47.202370 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.202325    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86456d9bbc-gxnhh"]
Oct 30 20:26:47.232635 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.232604    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02ea9da9-c742-4322-9f01-f0f7b8f184ef-tmp\") pod \"downloads-6dd789c8b7-x6wrc\" (UID: \"02ea9da9-c742-4322-9f01-f0f7b8f184ef\") " pod="openshift-console/downloads-6dd789c8b7-x6wrc"
Oct 30 20:26:47.232813 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.232651    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e06726d-3d3a-4934-9078-ff976aed6044-installation-pull-secrets\") pod \"image-registry-86456d9bbc-gxnhh\" (UID: \"2e06726d-3d3a-4934-9078-ff976aed6044\") " pod="openshift-image-registry/image-registry-86456d9bbc-gxnhh"
Oct 30 20:26:47.232813 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.232681    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8skh\" (UniqueName: \"kubernetes.io/projected/02ea9da9-c742-4322-9f01-f0f7b8f184ef-kube-api-access-m8skh\") pod \"downloads-6dd789c8b7-x6wrc\" (UID: \"02ea9da9-c742-4322-9f01-f0f7b8f184ef\") " pod="openshift-console/downloads-6dd789c8b7-x6wrc"
Oct 30 20:26:47.232813 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.232714    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e06726d-3d3a-4934-9078-ff976aed6044-trusted-ca\") pod \"image-registry-86456d9bbc-gxnhh\" (UID: \"2e06726d-3d3a-4934-9078-ff976aed6044\") " pod="openshift-image-registry/image-registry-86456d9bbc-gxnhh"
Oct 30 20:26:47.232813 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.232738    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5986f\" (UniqueName: \"kubernetes.io/projected/2e06726d-3d3a-4934-9078-ff976aed6044-kube-api-access-5986f\") pod \"image-registry-86456d9bbc-gxnhh\" (UID: \"2e06726d-3d3a-4934-9078-ff976aed6044\") " pod="openshift-image-registry/image-registry-86456d9bbc-gxnhh"
Oct 30 20:26:47.232813 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.232796    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e06726d-3d3a-4934-9078-ff976aed6044-ca-trust-extracted\") pod \"image-registry-86456d9bbc-gxnhh\" (UID: \"2e06726d-3d3a-4934-9078-ff976aed6044\") " pod="openshift-image-registry/image-registry-86456d9bbc-gxnhh"
Oct 30 20:26:47.233074 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.232853    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e06726d-3d3a-4934-9078-ff976aed6044-registry-tls\") pod \"image-registry-86456d9bbc-gxnhh\" (UID: \"2e06726d-3d3a-4934-9078-ff976aed6044\") " pod="openshift-image-registry/image-registry-86456d9bbc-gxnhh"
Oct 30 20:26:47.233074 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.232882    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e06726d-3d3a-4934-9078-ff976aed6044-image-registry-private-configuration\") pod \"image-registry-86456d9bbc-gxnhh\" (UID: \"2e06726d-3d3a-4934-9078-ff976aed6044\") " pod="openshift-image-registry/image-registry-86456d9bbc-gxnhh"
Oct 30 20:26:47.233074 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.232926    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e06726d-3d3a-4934-9078-ff976aed6044-registry-certificates\") pod \"image-registry-86456d9bbc-gxnhh\" (UID: \"2e06726d-3d3a-4934-9078-ff976aed6044\") " pod="openshift-image-registry/image-registry-86456d9bbc-gxnhh"
Oct 30 20:26:47.233074 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.232954    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e06726d-3d3a-4934-9078-ff976aed6044-bound-sa-token\") pod \"image-registry-86456d9bbc-gxnhh\" (UID: \"2e06726d-3d3a-4934-9078-ff976aed6044\") " pod="openshift-image-registry/image-registry-86456d9bbc-gxnhh"
Oct 30 20:26:47.233363 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.233336    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02ea9da9-c742-4322-9f01-f0f7b8f184ef-tmp\") pod \"downloads-6dd789c8b7-x6wrc\" (UID: \"02ea9da9-c742-4322-9f01-f0f7b8f184ef\") " pod="openshift-console/downloads-6dd789c8b7-x6wrc"
Oct 30 20:26:47.295105 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.295081    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8skh\" (UniqueName: \"kubernetes.io/projected/02ea9da9-c742-4322-9f01-f0f7b8f184ef-kube-api-access-m8skh\") pod \"downloads-6dd789c8b7-x6wrc\" (UID: \"02ea9da9-c742-4322-9f01-f0f7b8f184ef\") " pod="openshift-console/downloads-6dd789c8b7-x6wrc"
Oct 30 20:26:47.333515 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.333492    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e06726d-3d3a-4934-9078-ff976aed6044-registry-certificates\") pod \"image-registry-86456d9bbc-gxnhh\" (UID: \"2e06726d-3d3a-4934-9078-ff976aed6044\") " pod="openshift-image-registry/image-registry-86456d9bbc-gxnhh"
Oct 30 20:26:47.333654 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.333530    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e06726d-3d3a-4934-9078-ff976aed6044-bound-sa-token\") pod \"image-registry-86456d9bbc-gxnhh\" (UID: \"2e06726d-3d3a-4934-9078-ff976aed6044\") " pod="openshift-image-registry/image-registry-86456d9bbc-gxnhh"
Oct 30 20:26:47.333724 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.333663    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e06726d-3d3a-4934-9078-ff976aed6044-installation-pull-secrets\") pod \"image-registry-86456d9bbc-gxnhh\" (UID: \"2e06726d-3d3a-4934-9078-ff976aed6044\") " pod="openshift-image-registry/image-registry-86456d9bbc-gxnhh"
Oct 30 20:26:47.333724 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.333715    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e06726d-3d3a-4934-9078-ff976aed6044-trusted-ca\") pod \"image-registry-86456d9bbc-gxnhh\" (UID: \"2e06726d-3d3a-4934-9078-ff976aed6044\") " pod="openshift-image-registry/image-registry-86456d9bbc-gxnhh"
Oct 30 20:26:47.333866 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.333743    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5986f\" (UniqueName: \"kubernetes.io/projected/2e06726d-3d3a-4934-9078-ff976aed6044-kube-api-access-5986f\") pod \"image-registry-86456d9bbc-gxnhh\" (UID: \"2e06726d-3d3a-4934-9078-ff976aed6044\") " pod="openshift-image-registry/image-registry-86456d9bbc-gxnhh"
Oct 30 20:26:47.333866 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.333834    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e06726d-3d3a-4934-9078-ff976aed6044-ca-trust-extracted\") pod \"image-registry-86456d9bbc-gxnhh\" (UID: \"2e06726d-3d3a-4934-9078-ff976aed6044\") " pod="openshift-image-registry/image-registry-86456d9bbc-gxnhh"
Oct 30 20:26:47.333982 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.333876    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e06726d-3d3a-4934-9078-ff976aed6044-registry-tls\") pod \"image-registry-86456d9bbc-gxnhh\" (UID: \"2e06726d-3d3a-4934-9078-ff976aed6044\") " pod="openshift-image-registry/image-registry-86456d9bbc-gxnhh"
Oct 30 20:26:47.333982 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.333904    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e06726d-3d3a-4934-9078-ff976aed6044-image-registry-private-configuration\") pod \"image-registry-86456d9bbc-gxnhh\" (UID: \"2e06726d-3d3a-4934-9078-ff976aed6044\") " pod="openshift-image-registry/image-registry-86456d9bbc-gxnhh"
Oct 30 20:26:47.334218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.334164    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e06726d-3d3a-4934-9078-ff976aed6044-ca-trust-extracted\") pod \"image-registry-86456d9bbc-gxnhh\" (UID: \"2e06726d-3d3a-4934-9078-ff976aed6044\") " pod="openshift-image-registry/image-registry-86456d9bbc-gxnhh"
Oct 30 20:26:47.334601 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.334456    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e06726d-3d3a-4934-9078-ff976aed6044-registry-certificates\") pod \"image-registry-86456d9bbc-gxnhh\" (UID: \"2e06726d-3d3a-4934-9078-ff976aed6044\") " pod="openshift-image-registry/image-registry-86456d9bbc-gxnhh"
Oct 30 20:26:47.334925 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.334905    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e06726d-3d3a-4934-9078-ff976aed6044-trusted-ca\") pod \"image-registry-86456d9bbc-gxnhh\" (UID: \"2e06726d-3d3a-4934-9078-ff976aed6044\") " pod="openshift-image-registry/image-registry-86456d9bbc-gxnhh"
Oct 30 20:26:47.336455 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.336385    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e06726d-3d3a-4934-9078-ff976aed6044-installation-pull-secrets\") pod \"image-registry-86456d9bbc-gxnhh\" (UID: \"2e06726d-3d3a-4934-9078-ff976aed6044\") " pod="openshift-image-registry/image-registry-86456d9bbc-gxnhh"
Oct 30 20:26:47.336455 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.336412    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-x6wrc"
Oct 30 20:26:47.336680 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.336656    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e06726d-3d3a-4934-9078-ff976aed6044-image-registry-private-configuration\") pod \"image-registry-86456d9bbc-gxnhh\" (UID: \"2e06726d-3d3a-4934-9078-ff976aed6044\") " pod="openshift-image-registry/image-registry-86456d9bbc-gxnhh"
Oct 30 20:26:47.337177 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.337157    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e06726d-3d3a-4934-9078-ff976aed6044-registry-tls\") pod \"image-registry-86456d9bbc-gxnhh\" (UID: \"2e06726d-3d3a-4934-9078-ff976aed6044\") " pod="openshift-image-registry/image-registry-86456d9bbc-gxnhh"
Oct 30 20:26:47.385555 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.385524    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tk6nb" event={"ID":"b87eabe7-2403-4534-987b-ce6ecc6fda94","Type":"ContainerStarted","Data":"ffe2638c9aa13b013d5cefc41e11f591907dc490e0a4fd7055f82f2a16839a60"}
Oct 30 20:26:47.398782 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.398759    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e06726d-3d3a-4934-9078-ff976aed6044-bound-sa-token\") pod \"image-registry-86456d9bbc-gxnhh\" (UID: \"2e06726d-3d3a-4934-9078-ff976aed6044\") " pod="openshift-image-registry/image-registry-86456d9bbc-gxnhh"
Oct 30 20:26:47.404038 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.404014    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5986f\" (UniqueName: \"kubernetes.io/projected/2e06726d-3d3a-4934-9078-ff976aed6044-kube-api-access-5986f\") pod \"image-registry-86456d9bbc-gxnhh\" (UID: \"2e06726d-3d3a-4934-9078-ff976aed6044\") " pod="openshift-image-registry/image-registry-86456d9bbc-gxnhh"
Oct 30 20:26:47.428344 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.428312    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86456d9bbc-gxnhh"
Oct 30 20:26:47.542039 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.542007    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:47.576194 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.576167    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:47.766129 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.765294    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-x6wrc"]
Oct 30 20:26:47.767657 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:47.767624    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02ea9da9_c742_4322_9f01_f0f7b8f184ef.slice/crio-24e4ceb6d1ac1d7d0f95ba1fcfbfcb8b1887fbf14716340f1c1202f1f028ebe6 WatchSource:0}: Error finding container 24e4ceb6d1ac1d7d0f95ba1fcfbfcb8b1887fbf14716340f1c1202f1f028ebe6: Status 404 returned error can't find the container with id 24e4ceb6d1ac1d7d0f95ba1fcfbfcb8b1887fbf14716340f1c1202f1f028ebe6
Oct 30 20:26:47.779588 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.779556    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:47.819059 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:47.818986    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86456d9bbc-gxnhh"]
Oct 30 20:26:47.822151 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:26:47.822111    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e06726d_3d3a_4934_9078_ff976aed6044.slice/crio-e07ee02411b3c54470f8578d4b76b86a24c6b19f13bfb1bf4919ad14851f4898 WatchSource:0}: Error finding container e07ee02411b3c54470f8578d4b76b86a24c6b19f13bfb1bf4919ad14851f4898: Status 404 returned error can't find the container with id e07ee02411b3c54470f8578d4b76b86a24c6b19f13bfb1bf4919ad14851f4898
Oct 30 20:26:48.049315 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:48.049151    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:48.400374 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:48.400339    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-x6wrc" event={"ID":"02ea9da9-c742-4322-9f01-f0f7b8f184ef","Type":"ContainerStarted","Data":"24e4ceb6d1ac1d7d0f95ba1fcfbfcb8b1887fbf14716340f1c1202f1f028ebe6"}
Oct 30 20:26:48.402901 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:48.402871    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86456d9bbc-gxnhh" event={"ID":"2e06726d-3d3a-4934-9078-ff976aed6044","Type":"ContainerStarted","Data":"fd43210187c64c6c98acd3575191fbfe18852f9967f40a178e06d84c9336a9e5"}
Oct 30 20:26:48.403013 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:48.402933    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:48.403013 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:48.402956    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-86456d9bbc-gxnhh"
Oct 30 20:26:48.403013 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:48.402974    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86456d9bbc-gxnhh" event={"ID":"2e06726d-3d3a-4934-9078-ff976aed6044","Type":"ContainerStarted","Data":"e07ee02411b3c54470f8578d4b76b86a24c6b19f13bfb1bf4919ad14851f4898"}
Oct 30 20:26:48.405316 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:48.405290    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-666dd86ff-dnnq5"
Oct 30 20:26:48.525572 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:48.525514    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-86456d9bbc-gxnhh" podStartSLOduration=1.5254966319999999 podStartE2EDuration="1.525496632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:48.524295086 +0000 UTC m=+20.441316622" watchObservedRunningTime="2026-10-30 20:26:48.525496632 +0000 UTC m=+20.442518170"
Oct 30 20:26:49.409417 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:49.409380    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-9rs22" event={"ID":"d5312b70-82df-4e75-a300-b8e1d63a75f3","Type":"ContainerStarted","Data":"a512f5207b6223209a77953809896b58419480b2842262c3b8c10c85b3f57c44"}
Oct 30 20:26:49.411762 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:49.411735    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tk6nb" event={"ID":"b87eabe7-2403-4534-987b-ce6ecc6fda94","Type":"ContainerStarted","Data":"26e7dae615266517bfda4579a2555aee6bdbb07187f10ed98992ba986b97396d"}
Oct 30 20:26:49.742536 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:49.742487    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:50.007939 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:50.007852    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:50.688450 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:50.688414    5071 patch_prober.go:29] interesting pod/image-registry-7db8fb6d4-8n6g7 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 20:26:50.689016 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:50.688472    5071 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7" podUID="05f22644-0884-434d-b56b-193d9554b505" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:26:51.001033 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:51.000954    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:51.190698 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:51.190650    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:51.358093 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:51.358041    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-h2vh7"
Oct 30 20:26:53.528184 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:53.528145    5071 patch_prober.go:29] interesting pod/image-registry-7db8fb6d4-8n6g7 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 20:26:53.528634 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:53.528208    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7" podUID="05f22644-0884-434d-b56b-193d9554b505" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:26:54.161085 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:54.161048    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:54.369398 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:54.369362    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:54.573357 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:54.573302    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:54.744454 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:54.744419    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:57.037399 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:57.037363    5071 patch_prober.go:29] interesting pod/image-registry-968d68cdf-vwtgc container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 20:26:57.037845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:57.037421    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-968d68cdf-vwtgc" podUID="9f908129-4500-4433-871c-032c29a55dd9" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:26:57.384844 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:57.384799    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pfvnn"
Oct 30 20:26:59.344357 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:59.344317    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:26:59.571654 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:26:59.571621    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:00.353923 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:00.353746    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:00.570120 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:00.570066    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:00.732624 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:00.732540    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:00.899468 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:00.899431    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:01.086295 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.086262    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:01.244699 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.244663    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mvkzt"]
Oct 30 20:27:01.274227 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.274190    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:01.281748 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.281721    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mvkzt"
Oct 30 20:27:01.357945 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.357874    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e21d7e79-1c77-4815-b43d-603881b3252c-node-exporter-textfile\") pod \"node-exporter-mvkzt\" (UID: \"e21d7e79-1c77-4815-b43d-603881b3252c\") " pod="openshift-monitoring/node-exporter-mvkzt"
Oct 30 20:27:01.357945 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.357910    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e21d7e79-1c77-4815-b43d-603881b3252c-node-exporter-accelerators-collector-config\") pod \"node-exporter-mvkzt\" (UID: \"e21d7e79-1c77-4815-b43d-603881b3252c\") " pod="openshift-monitoring/node-exporter-mvkzt"
Oct 30 20:27:01.357945 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.357938    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e21d7e79-1c77-4815-b43d-603881b3252c-sys\") pod \"node-exporter-mvkzt\" (UID: \"e21d7e79-1c77-4815-b43d-603881b3252c\") " pod="openshift-monitoring/node-exporter-mvkzt"
Oct 30 20:27:01.358559 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.358012    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e21d7e79-1c77-4815-b43d-603881b3252c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mvkzt\" (UID: \"e21d7e79-1c77-4815-b43d-603881b3252c\") " pod="openshift-monitoring/node-exporter-mvkzt"
Oct 30 20:27:01.358559 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.358040    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhrxn\" (UniqueName: \"kubernetes.io/projected/e21d7e79-1c77-4815-b43d-603881b3252c-kube-api-access-jhrxn\") pod \"node-exporter-mvkzt\" (UID: \"e21d7e79-1c77-4815-b43d-603881b3252c\") " pod="openshift-monitoring/node-exporter-mvkzt"
Oct 30 20:27:01.358559 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.358065    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e21d7e79-1c77-4815-b43d-603881b3252c-root\") pod \"node-exporter-mvkzt\" (UID: \"e21d7e79-1c77-4815-b43d-603881b3252c\") " pod="openshift-monitoring/node-exporter-mvkzt"
Oct 30 20:27:01.358559 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.358089    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e21d7e79-1c77-4815-b43d-603881b3252c-node-exporter-wtmp\") pod \"node-exporter-mvkzt\" (UID: \"e21d7e79-1c77-4815-b43d-603881b3252c\") " pod="openshift-monitoring/node-exporter-mvkzt"
Oct 30 20:27:01.358559 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.358139    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e21d7e79-1c77-4815-b43d-603881b3252c-node-exporter-tls\") pod \"node-exporter-mvkzt\" (UID: \"e21d7e79-1c77-4815-b43d-603881b3252c\") " pod="openshift-monitoring/node-exporter-mvkzt"
Oct 30 20:27:01.358559 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.358178    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e21d7e79-1c77-4815-b43d-603881b3252c-metrics-client-ca\") pod \"node-exporter-mvkzt\" (UID: \"e21d7e79-1c77-4815-b43d-603881b3252c\") " pod="openshift-monitoring/node-exporter-mvkzt"
Oct 30 20:27:01.386417 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.386385    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 20:27:01.386574 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.386470    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 20:27:01.435901 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.435861    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 20:27:01.435901 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.435891    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 20:27:01.436147 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.435931    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nwr4g\""
Oct 30 20:27:01.460110 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.460085    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e21d7e79-1c77-4815-b43d-603881b3252c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mvkzt\" (UID: \"e21d7e79-1c77-4815-b43d-603881b3252c\") " pod="openshift-monitoring/node-exporter-mvkzt"
Oct 30 20:27:01.460110 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.460112    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhrxn\" (UniqueName: \"kubernetes.io/projected/e21d7e79-1c77-4815-b43d-603881b3252c-kube-api-access-jhrxn\") pod \"node-exporter-mvkzt\" (UID: \"e21d7e79-1c77-4815-b43d-603881b3252c\") " pod="openshift-monitoring/node-exporter-mvkzt"
Oct 30 20:27:01.460330 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.460129    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e21d7e79-1c77-4815-b43d-603881b3252c-root\") pod \"node-exporter-mvkzt\" (UID: \"e21d7e79-1c77-4815-b43d-603881b3252c\") " pod="openshift-monitoring/node-exporter-mvkzt"
Oct 30 20:27:01.460330 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.460145    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e21d7e79-1c77-4815-b43d-603881b3252c-node-exporter-wtmp\") pod \"node-exporter-mvkzt\" (UID: \"e21d7e79-1c77-4815-b43d-603881b3252c\") " pod="openshift-monitoring/node-exporter-mvkzt"
Oct 30 20:27:01.460330 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.460190    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e21d7e79-1c77-4815-b43d-603881b3252c-node-exporter-tls\") pod \"node-exporter-mvkzt\" (UID: \"e21d7e79-1c77-4815-b43d-603881b3252c\") " pod="openshift-monitoring/node-exporter-mvkzt"
Oct 30 20:27:01.460330 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.460217    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e21d7e79-1c77-4815-b43d-603881b3252c-root\") pod \"node-exporter-mvkzt\" (UID: \"e21d7e79-1c77-4815-b43d-603881b3252c\") " pod="openshift-monitoring/node-exporter-mvkzt"
Oct 30 20:27:01.460330 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.460227    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e21d7e79-1c77-4815-b43d-603881b3252c-metrics-client-ca\") pod \"node-exporter-mvkzt\" (UID: \"e21d7e79-1c77-4815-b43d-603881b3252c\") " pod="openshift-monitoring/node-exporter-mvkzt"
Oct 30 20:27:01.460330 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.460286    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e21d7e79-1c77-4815-b43d-603881b3252c-node-exporter-textfile\") pod \"node-exporter-mvkzt\" (UID: \"e21d7e79-1c77-4815-b43d-603881b3252c\") " pod="openshift-monitoring/node-exporter-mvkzt"
Oct 30 20:27:01.460330 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.460309    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e21d7e79-1c77-4815-b43d-603881b3252c-node-exporter-accelerators-collector-config\") pod \"node-exporter-mvkzt\" (UID: \"e21d7e79-1c77-4815-b43d-603881b3252c\") " pod="openshift-monitoring/node-exporter-mvkzt"
Oct 30 20:27:01.460670 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.460340    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e21d7e79-1c77-4815-b43d-603881b3252c-sys\") pod \"node-exporter-mvkzt\" (UID: \"e21d7e79-1c77-4815-b43d-603881b3252c\") " pod="openshift-monitoring/node-exporter-mvkzt"
Oct 30 20:27:01.460670 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.460420    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e21d7e79-1c77-4815-b43d-603881b3252c-sys\") pod \"node-exporter-mvkzt\" (UID: \"e21d7e79-1c77-4815-b43d-603881b3252c\") " pod="openshift-monitoring/node-exporter-mvkzt"
Oct 30 20:27:01.460670 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.460515    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e21d7e79-1c77-4815-b43d-603881b3252c-node-exporter-wtmp\") pod \"node-exporter-mvkzt\" (UID: \"e21d7e79-1c77-4815-b43d-603881b3252c\") " pod="openshift-monitoring/node-exporter-mvkzt"
Oct 30 20:27:01.460768 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.460698    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e21d7e79-1c77-4815-b43d-603881b3252c-node-exporter-textfile\") pod \"node-exporter-mvkzt\" (UID: \"e21d7e79-1c77-4815-b43d-603881b3252c\") " pod="openshift-monitoring/node-exporter-mvkzt"
Oct 30 20:27:01.460802 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.460769    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e21d7e79-1c77-4815-b43d-603881b3252c-metrics-client-ca\") pod \"node-exporter-mvkzt\" (UID: \"e21d7e79-1c77-4815-b43d-603881b3252c\") " pod="openshift-monitoring/node-exporter-mvkzt"
Oct 30 20:27:01.461000 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.460982    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e21d7e79-1c77-4815-b43d-603881b3252c-node-exporter-accelerators-collector-config\") pod \"node-exporter-mvkzt\" (UID: \"e21d7e79-1c77-4815-b43d-603881b3252c\") " pod="openshift-monitoring/node-exporter-mvkzt"
Oct 30 20:27:01.462920 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.462893    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e21d7e79-1c77-4815-b43d-603881b3252c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mvkzt\" (UID: \"e21d7e79-1c77-4815-b43d-603881b3252c\") " pod="openshift-monitoring/node-exporter-mvkzt"
Oct 30 20:27:01.463095 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.463075    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e21d7e79-1c77-4815-b43d-603881b3252c-node-exporter-tls\") pod \"node-exporter-mvkzt\" (UID: \"e21d7e79-1c77-4815-b43d-603881b3252c\") " pod="openshift-monitoring/node-exporter-mvkzt"
Oct 30 20:27:01.516490 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.516429    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:01.539806 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.539778    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhrxn\" (UniqueName: \"kubernetes.io/projected/e21d7e79-1c77-4815-b43d-603881b3252c-kube-api-access-jhrxn\") pod \"node-exporter-mvkzt\" (UID: \"e21d7e79-1c77-4815-b43d-603881b3252c\") " pod="openshift-monitoring/node-exporter-mvkzt"
Oct 30 20:27:01.640891 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.636985    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mvkzt"
Oct 30 20:27:01.686797 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:27:01.686766    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode21d7e79_1c77_4815_b43d_603881b3252c.slice/crio-6651159af73231ac1dbeec7d287566edec4ecee2eb2857a44f74662c24dfcc55 WatchSource:0}: Error finding container 6651159af73231ac1dbeec7d287566edec4ecee2eb2857a44f74662c24dfcc55: Status 404 returned error can't find the container with id 6651159af73231ac1dbeec7d287566edec4ecee2eb2857a44f74662c24dfcc55
Oct 30 20:27:01.772500 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.772464    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:01.949542 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:01.949427    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:02.474105 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:02.474068    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mvkzt" event={"ID":"e21d7e79-1c77-4815-b43d-603881b3252c","Type":"ContainerStarted","Data":"6651159af73231ac1dbeec7d287566edec4ecee2eb2857a44f74662c24dfcc55"}
Oct 30 20:27:02.607656 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:02.607620    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:02.799520 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:02.799475    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:03.528219 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:03.528191    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:27:03.533390 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:03.533359    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:04.273414 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:04.273376    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:04.470339 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:04.470244    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:04.485314 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:04.485275    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-x6wrc" event={"ID":"02ea9da9-c742-4322-9f01-f0f7b8f184ef","Type":"ContainerStarted","Data":"7ad54b18698481b16b4e2b357228a74ce58e6e34655b61d9e032754b60a80725"}
Oct 30 20:27:04.485492 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:04.485350    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-x6wrc"
Oct 30 20:27:04.488339 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:04.488315    5071 generic.go:356] "Generic (PLEG): container finished" podID="e21d7e79-1c77-4815-b43d-603881b3252c" containerID="ff607b3bf8c08c067e926e6621eaae68f6c400f7982d1a8321b41739412986ad" exitCode=0
Oct 30 20:27:04.488458 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:04.488355    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mvkzt" event={"ID":"e21d7e79-1c77-4815-b43d-603881b3252c","Type":"ContainerDied","Data":"ff607b3bf8c08c067e926e6621eaae68f6c400f7982d1a8321b41739412986ad"}
Oct 30 20:27:04.506511 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:04.506443    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-x6wrc"
Oct 30 20:27:04.604190 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:04.603841    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-x6wrc" podStartSLOduration=2.250112156 podStartE2EDuration="18.603809534s" totalImagesPullingTime="16.353697378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:04.602290527 +0000 UTC m=+36.519312058" watchObservedRunningTime="2026-10-30 20:27:04.603809534 +0000 UTC m=+36.520831074"
Oct 30 20:27:04.637697 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:04.637665    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:04.816240 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:04.816205    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:05.497521 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:05.497477    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mvkzt" event={"ID":"e21d7e79-1c77-4815-b43d-603881b3252c","Type":"ContainerStarted","Data":"739b2cf66d6e29f2c1966fd81ff07c3d13754c2ba58145c85b91235a86827a10"}
Oct 30 20:27:05.497707 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:05.497546    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mvkzt" event={"ID":"e21d7e79-1c77-4815-b43d-603881b3252c","Type":"ContainerStarted","Data":"850d510d197151b79f2831933f0449ee8d4e894e0b63cc209b5adc6130b421a4"}
Oct 30 20:27:05.621049 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:05.620991    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mvkzt" podStartSLOduration=2.538367542 podStartE2EDuration="4.620973911s" totalImagesPullingTime="1.986798662s" totalInitContainerRuntime="95.807707ms" isStatefulPod=true podCreationTimestamp="2026-10-30 20:27:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:05.620219778 +0000 UTC m=+37.537241316" watchObservedRunningTime="2026-10-30 20:27:05.620973911 +0000 UTC m=+37.537995448"
Oct 30 20:27:05.663616 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:05.663539    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:05.914108 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:05.914071    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:06.076487 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:06.076444    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:06.353321 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:06.353232    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:06.539450 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:06.539410    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:07.034480 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:07.034451    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:27:07.768263 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:07.768216    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:07.940351 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:07.940311    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:09.416459 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:09.416428    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-86456d9bbc-gxnhh"
Oct 30 20:27:09.496809 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:09.496776    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:09.655413 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:09.655379    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7db8fb6d4-8n6g7"]
Oct 30 20:27:09.751994 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:09.751901    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:09.913251 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:09.913216    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:10.097494 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:10.097442    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:10.256192 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:10.256154    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:10.764454 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:10.764411    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:11.177538 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:11.177503    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:11.376748 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:11.376712    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:12.057180 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:12.057135    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-968d68cdf-vwtgc" podUID="9f908129-4500-4433-871c-032c29a55dd9" containerName="registry" containerID="cri-o://5c38e1613503f4bf4f84774a2192eda53162f86f31dd7d5aa81641c2e532a377" gracePeriod=30
Oct 30 20:27:12.350443 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:12.350419    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:27:12.463439 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:12.463404    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f908129-4500-4433-871c-032c29a55dd9-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9f908129-4500-4433-871c-032c29a55dd9-ca-trust-extracted\") pod \"9f908129-4500-4433-871c-032c29a55dd9\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") "
Oct 30 20:27:12.463617 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:12.463475    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f908129-4500-4433-871c-032c29a55dd9-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9f908129-4500-4433-871c-032c29a55dd9-registry-certificates\") pod \"9f908129-4500-4433-871c-032c29a55dd9\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") "
Oct 30 20:27:12.463617 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:12.463536    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f908129-4500-4433-871c-032c29a55dd9-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9f908129-4500-4433-871c-032c29a55dd9-image-registry-private-configuration\") pod \"9f908129-4500-4433-871c-032c29a55dd9\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") "
Oct 30 20:27:12.463617 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:12.463567    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-bound-sa-token\") pod \"9f908129-4500-4433-871c-032c29a55dd9\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") "
Oct 30 20:27:12.463772 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:12.463622    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-registry-tls\" (UniqueName: \"kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-registry-tls\") pod \"9f908129-4500-4433-871c-032c29a55dd9\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") "
Oct 30 20:27:12.463772 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:12.463710    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-kube-api-access-nqftg\" (UniqueName: \"kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-kube-api-access-nqftg\") pod \"9f908129-4500-4433-871c-032c29a55dd9\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") "
Oct 30 20:27:12.463911 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:12.463775    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f908129-4500-4433-871c-032c29a55dd9-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f908129-4500-4433-871c-032c29a55dd9-trusted-ca\") pod \"9f908129-4500-4433-871c-032c29a55dd9\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") "
Oct 30 20:27:12.463911 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:12.463844    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f908129-4500-4433-871c-032c29a55dd9-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9f908129-4500-4433-871c-032c29a55dd9-installation-pull-secrets\") pod \"9f908129-4500-4433-871c-032c29a55dd9\" (UID: \"9f908129-4500-4433-871c-032c29a55dd9\") "
Oct 30 20:27:12.467429 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:12.467374    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f908129-4500-4433-871c-032c29a55dd9-registry-certificates" pod "9f908129-4500-4433-871c-032c29a55dd9" (UID: "9f908129-4500-4433-871c-032c29a55dd9"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:27:12.467562 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:12.467426    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f908129-4500-4433-871c-032c29a55dd9-trusted-ca" pod "9f908129-4500-4433-871c-032c29a55dd9" (UID: "9f908129-4500-4433-871c-032c29a55dd9"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:27:12.470750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:12.470346    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-kube-api-access-nqftg" pod "9f908129-4500-4433-871c-032c29a55dd9" (UID: "9f908129-4500-4433-871c-032c29a55dd9"). InnerVolumeSpecName "kube-api-access-nqftg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:27:12.470750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:12.470443    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f908129-4500-4433-871c-032c29a55dd9-image-registry-private-configuration" pod "9f908129-4500-4433-871c-032c29a55dd9" (UID: "9f908129-4500-4433-871c-032c29a55dd9"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:27:12.470750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:12.470750    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-bound-sa-token" pod "9f908129-4500-4433-871c-032c29a55dd9" (UID: "9f908129-4500-4433-871c-032c29a55dd9"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:27:12.474311 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:12.474236    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f908129-4500-4433-871c-032c29a55dd9-ca-trust-extracted" pod "9f908129-4500-4433-871c-032c29a55dd9" (UID: "9f908129-4500-4433-871c-032c29a55dd9"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:27:12.478296 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:12.478216    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-registry-tls" pod "9f908129-4500-4433-871c-032c29a55dd9" (UID: "9f908129-4500-4433-871c-032c29a55dd9"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:27:12.478296 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:12.478271    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f908129-4500-4433-871c-032c29a55dd9-installation-pull-secrets" pod "9f908129-4500-4433-871c-032c29a55dd9" (UID: "9f908129-4500-4433-871c-032c29a55dd9"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:27:12.532443 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:12.532403    5071 generic.go:356] "Generic (PLEG): container finished" podID="9f908129-4500-4433-871c-032c29a55dd9" containerID="5c38e1613503f4bf4f84774a2192eda53162f86f31dd7d5aa81641c2e532a377" exitCode=0
Oct 30 20:27:12.532691 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:12.532478    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-968d68cdf-vwtgc" event={"ID":"9f908129-4500-4433-871c-032c29a55dd9","Type":"ContainerDied","Data":"5c38e1613503f4bf4f84774a2192eda53162f86f31dd7d5aa81641c2e532a377"}
Oct 30 20:27:12.532691 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:12.532581    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-968d68cdf-vwtgc" event={"ID":"9f908129-4500-4433-871c-032c29a55dd9","Type":"ContainerDied","Data":"a264ea4a1ac8f9f1b74cc603b8b44e7b059fb275cf1c1fc56bfdc5efca7b7a7a"}
Oct 30 20:27:12.532691 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:12.532617    5071 scope.go:122] "RemoveContainer" containerID="5c38e1613503f4bf4f84774a2192eda53162f86f31dd7d5aa81641c2e532a377"
Oct 30 20:27:12.532918 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:12.532769    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-968d68cdf-vwtgc"
Oct 30 20:27:12.549938 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:12.549893    5071 scope.go:122] "RemoveContainer" containerID="5c38e1613503f4bf4f84774a2192eda53162f86f31dd7d5aa81641c2e532a377"
Oct 30 20:27:12.550575 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:27:12.550503    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c38e1613503f4bf4f84774a2192eda53162f86f31dd7d5aa81641c2e532a377\": container with ID starting with 5c38e1613503f4bf4f84774a2192eda53162f86f31dd7d5aa81641c2e532a377 not found: ID does not exist" containerID="5c38e1613503f4bf4f84774a2192eda53162f86f31dd7d5aa81641c2e532a377"
Oct 30 20:27:12.550686 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:12.550572    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c38e1613503f4bf4f84774a2192eda53162f86f31dd7d5aa81641c2e532a377"} err="failed to get container status \"5c38e1613503f4bf4f84774a2192eda53162f86f31dd7d5aa81641c2e532a377\": rpc error: code = NotFound desc = could not find container \"5c38e1613503f4bf4f84774a2192eda53162f86f31dd7d5aa81641c2e532a377\": container with ID starting with 5c38e1613503f4bf4f84774a2192eda53162f86f31dd7d5aa81641c2e532a377 not found: ID does not exist"
Oct 30 20:27:12.565224 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:12.565140    5071 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-registry-tls\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:12.565224 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:12.565174    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqftg\" (UniqueName: \"kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-kube-api-access-nqftg\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:12.565224 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:12.565192    5071 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f908129-4500-4433-871c-032c29a55dd9-trusted-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:12.565224 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:12.565207    5071 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9f908129-4500-4433-871c-032c29a55dd9-installation-pull-secrets\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:12.565224 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:12.565221    5071 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9f908129-4500-4433-871c-032c29a55dd9-ca-trust-extracted\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:12.565506 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:12.565235    5071 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9f908129-4500-4433-871c-032c29a55dd9-registry-certificates\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:12.565506 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:12.565251    5071 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9f908129-4500-4433-871c-032c29a55dd9-image-registry-private-configuration\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:12.565506 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:12.565266    5071 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f908129-4500-4433-871c-032c29a55dd9-bound-sa-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:12.702243 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:12.702200    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-968d68cdf-vwtgc"]
Oct 30 20:27:12.713468 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:12.713434    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-968d68cdf-vwtgc"]
Oct 30 20:27:12.772141 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:12.772101    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:12.968688 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:12.968607    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:13.928081 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:13.928043    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:14.103048 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:14.103013    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:14.199448 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:14.199380    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f908129-4500-4433-871c-032c29a55dd9" path="/var/lib/kubelet/pods/9f908129-4500-4433-871c-032c29a55dd9/volumes"
Oct 30 20:27:15.549651 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:15.549618    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:15.717473 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:15.717426    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:16.727095 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:27:16.727051    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:27:17.855266 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:17.855229    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:18.045398 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:18.045368    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:18.485270 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:18.485237    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:18.725808 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:18.725778    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:19.562242 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:19.562205    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:19.756766 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:19.756736    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:23.463485 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:23.463446    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:23.622281 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:23.622241    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:24.503640 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:24.503602    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:24.711361 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:24.711323    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:28.920307 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:28.920270    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:29.104874 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:29.104814    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:30.451997 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:30.451956    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:30.668544 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:30.668505    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:33.539163 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:33.539130    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:33.746866 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:33.746809    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:34.683490 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:34.683451    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7" podUID="05f22644-0884-434d-b56b-193d9554b505" containerName="registry" containerID="cri-o://14edc385f1899eecce2b388d841c8ca7dcb619a255508477ab4968ec0efd1811" gracePeriod=30
Oct 30 20:27:34.966170 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:34.966109    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:27:35.025623 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:35.025591    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-bound-sa-token\") pod \"05f22644-0884-434d-b56b-193d9554b505\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") "
Oct 30 20:27:35.025623 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:35.025628    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05f22644-0884-434d-b56b-193d9554b505-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/05f22644-0884-434d-b56b-193d9554b505-registry-certificates\") pod \"05f22644-0884-434d-b56b-193d9554b505\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") "
Oct 30 20:27:35.025929 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:35.025650    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05f22644-0884-434d-b56b-193d9554b505-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/05f22644-0884-434d-b56b-193d9554b505-installation-pull-secrets\") pod \"05f22644-0884-434d-b56b-193d9554b505\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") "
Oct 30 20:27:35.025929 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:35.025682    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f22644-0884-434d-b56b-193d9554b505-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/05f22644-0884-434d-b56b-193d9554b505-ca-trust-extracted\") pod \"05f22644-0884-434d-b56b-193d9554b505\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") "
Oct 30 20:27:35.025929 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:35.025721    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05f22644-0884-434d-b56b-193d9554b505-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/05f22644-0884-434d-b56b-193d9554b505-image-registry-private-configuration\") pod \"05f22644-0884-434d-b56b-193d9554b505\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") "
Oct 30 20:27:35.025929 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:35.025778    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-registry-tls\" (UniqueName: \"kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-registry-tls\") pod \"05f22644-0884-434d-b56b-193d9554b505\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") "
Oct 30 20:27:35.025929 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:35.025809    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05f22644-0884-434d-b56b-193d9554b505-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/05f22644-0884-434d-b56b-193d9554b505-trusted-ca\") pod \"05f22644-0884-434d-b56b-193d9554b505\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") "
Oct 30 20:27:35.025929 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:35.025852    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-kube-api-access-qh4r9\" (UniqueName: \"kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-kube-api-access-qh4r9\") pod \"05f22644-0884-434d-b56b-193d9554b505\" (UID: \"05f22644-0884-434d-b56b-193d9554b505\") "
Oct 30 20:27:35.026342 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:35.026306    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05f22644-0884-434d-b56b-193d9554b505-registry-certificates" pod "05f22644-0884-434d-b56b-193d9554b505" (UID: "05f22644-0884-434d-b56b-193d9554b505"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:27:35.026470 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:35.026346    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05f22644-0884-434d-b56b-193d9554b505-trusted-ca" pod "05f22644-0884-434d-b56b-193d9554b505" (UID: "05f22644-0884-434d-b56b-193d9554b505"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:27:35.028808 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:35.028552    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05f22644-0884-434d-b56b-193d9554b505-installation-pull-secrets" pod "05f22644-0884-434d-b56b-193d9554b505" (UID: "05f22644-0884-434d-b56b-193d9554b505"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:27:35.028808 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:35.028757    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-registry-tls" pod "05f22644-0884-434d-b56b-193d9554b505" (UID: "05f22644-0884-434d-b56b-193d9554b505"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:27:35.028808 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:35.028797    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-bound-sa-token" pod "05f22644-0884-434d-b56b-193d9554b505" (UID: "05f22644-0884-434d-b56b-193d9554b505"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:27:35.029016 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:35.028990    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05f22644-0884-434d-b56b-193d9554b505-image-registry-private-configuration" pod "05f22644-0884-434d-b56b-193d9554b505" (UID: "05f22644-0884-434d-b56b-193d9554b505"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:27:35.029016 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:35.029007    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-kube-api-access-qh4r9" pod "05f22644-0884-434d-b56b-193d9554b505" (UID: "05f22644-0884-434d-b56b-193d9554b505"). InnerVolumeSpecName "kube-api-access-qh4r9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:27:35.034414 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:35.034384    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f22644-0884-434d-b56b-193d9554b505-ca-trust-extracted" pod "05f22644-0884-434d-b56b-193d9554b505" (UID: "05f22644-0884-434d-b56b-193d9554b505"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:27:35.126341 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:35.126306    5071 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/05f22644-0884-434d-b56b-193d9554b505-installation-pull-secrets\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:35.126341 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:35.126334    5071 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/05f22644-0884-434d-b56b-193d9554b505-ca-trust-extracted\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:35.126341 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:35.126345    5071 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/05f22644-0884-434d-b56b-193d9554b505-image-registry-private-configuration\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:35.126616 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:35.126356    5071 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-registry-tls\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:35.126616 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:35.126365    5071 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/05f22644-0884-434d-b56b-193d9554b505-trusted-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:35.126616 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:35.126374    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh4r9\" (UniqueName: \"kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-kube-api-access-qh4r9\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:35.126616 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:35.126382    5071 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/05f22644-0884-434d-b56b-193d9554b505-bound-sa-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:35.126616 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:35.126391    5071 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/05f22644-0884-434d-b56b-193d9554b505-registry-certificates\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:35.641533 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:35.641498    5071 generic.go:356] "Generic (PLEG): container finished" podID="05f22644-0884-434d-b56b-193d9554b505" containerID="14edc385f1899eecce2b388d841c8ca7dcb619a255508477ab4968ec0efd1811" exitCode=0
Oct 30 20:27:35.641769 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:35.641564    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7"
Oct 30 20:27:35.641769 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:35.641611    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7" event={"ID":"05f22644-0884-434d-b56b-193d9554b505","Type":"ContainerDied","Data":"14edc385f1899eecce2b388d841c8ca7dcb619a255508477ab4968ec0efd1811"}
Oct 30 20:27:35.641769 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:35.641675    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7db8fb6d4-8n6g7" event={"ID":"05f22644-0884-434d-b56b-193d9554b505","Type":"ContainerDied","Data":"5e72fbd9e0a01d8fc01f3e5098fc29f32902db1d36ca6609e1599f8daa23a78c"}
Oct 30 20:27:35.641769 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:35.641702    5071 scope.go:122] "RemoveContainer" containerID="14edc385f1899eecce2b388d841c8ca7dcb619a255508477ab4968ec0efd1811"
Oct 30 20:27:35.660355 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:35.660338    5071 scope.go:122] "RemoveContainer" containerID="14edc385f1899eecce2b388d841c8ca7dcb619a255508477ab4968ec0efd1811"
Oct 30 20:27:35.660608 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:27:35.660590    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14edc385f1899eecce2b388d841c8ca7dcb619a255508477ab4968ec0efd1811\": container with ID starting with 14edc385f1899eecce2b388d841c8ca7dcb619a255508477ab4968ec0efd1811 not found: ID does not exist" containerID="14edc385f1899eecce2b388d841c8ca7dcb619a255508477ab4968ec0efd1811"
Oct 30 20:27:35.660684 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:35.660618    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14edc385f1899eecce2b388d841c8ca7dcb619a255508477ab4968ec0efd1811"} err="failed to get container status \"14edc385f1899eecce2b388d841c8ca7dcb619a255508477ab4968ec0efd1811\": rpc error: code = NotFound desc = could not find container \"14edc385f1899eecce2b388d841c8ca7dcb619a255508477ab4968ec0efd1811\": container with ID starting with 14edc385f1899eecce2b388d841c8ca7dcb619a255508477ab4968ec0efd1811 not found: ID does not exist"
Oct 30 20:27:35.808028 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:35.808002    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7db8fb6d4-8n6g7"]
Oct 30 20:27:35.858307 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:35.858276    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7db8fb6d4-8n6g7"]
Oct 30 20:27:36.199937 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:36.199899    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05f22644-0884-434d-b56b-193d9554b505" path="/var/lib/kubelet/pods/05f22644-0884-434d-b56b-193d9554b505/volumes"
Oct 30 20:27:38.505204 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:38.505172    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:38.726365 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:38.726330    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:38.927807 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:38.927767    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:39.438603 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:39.438562    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:39.596117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:39.596077    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:39.659636 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:39.659559    5071 generic.go:356] "Generic (PLEG): container finished" podID="8840f0ca-269e-4924-8b3d-5fe6ff4e90cf" containerID="6d6e8f3dc531658573169c482a26a7fcf917aec85fd5fc380ccd5a3d7c266197" exitCode=0
Oct 30 20:27:39.659789 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:39.659632    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-mql5m" event={"ID":"8840f0ca-269e-4924-8b3d-5fe6ff4e90cf","Type":"ContainerDied","Data":"6d6e8f3dc531658573169c482a26a7fcf917aec85fd5fc380ccd5a3d7c266197"}
Oct 30 20:27:39.659970 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:39.659953    5071 scope.go:122] "RemoveContainer" containerID="6d6e8f3dc531658573169c482a26a7fcf917aec85fd5fc380ccd5a3d7c266197"
Oct 30 20:27:40.665562 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:40.665526    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-mql5m" event={"ID":"8840f0ca-269e-4924-8b3d-5fe6ff4e90cf","Type":"ContainerStarted","Data":"dd30f15917c2f5c5f9765cad667ea09334ec11192b62b4a195b160496d84b39a"}
Oct 30 20:27:40.666893 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:40.666870    5071 generic.go:356] "Generic (PLEG): container finished" podID="5040db9b-91d3-4d89-adaa-d1394cc3a617" containerID="b7ecbaf4df570e688022c9d51b06a367bdbc0160489cf748651b668cb5d7e585" exitCode=0
Oct 30 20:27:40.667004 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:40.666940    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tp7sc" event={"ID":"5040db9b-91d3-4d89-adaa-d1394cc3a617","Type":"ContainerDied","Data":"b7ecbaf4df570e688022c9d51b06a367bdbc0160489cf748651b668cb5d7e585"}
Oct 30 20:27:40.667185 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:40.667171    5071 scope.go:122] "RemoveContainer" containerID="b7ecbaf4df570e688022c9d51b06a367bdbc0160489cf748651b668cb5d7e585"
Oct 30 20:27:40.770959 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:40.770920    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-zmknv-22np8"]
Oct 30 20:27:40.771153 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:40.771131    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-zmknv-22np8" podUID="ee380d52-b71c-4839-8835-078d98a5e0ee" containerName="insights-gathering" containerID="cri-o://d39f3158be11fb9a929222afc0f19562ec8f1dc76da6a005cce4cb2cb62eeb39" gracePeriod=30
Oct 30 20:27:41.059698 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:41.059673    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-zmknv-22np8"
Oct 30 20:27:41.068582 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:41.068560    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee380d52-b71c-4839-8835-078d98a5e0ee-kube-api-access-9l2sr\" (UniqueName: \"kubernetes.io/projected/ee380d52-b71c-4839-8835-078d98a5e0ee-kube-api-access-9l2sr\") pod \"ee380d52-b71c-4839-8835-078d98a5e0ee\" (UID: \"ee380d52-b71c-4839-8835-078d98a5e0ee\") "
Oct 30 20:27:41.068704 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:41.068602    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee380d52-b71c-4839-8835-078d98a5e0ee-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee380d52-b71c-4839-8835-078d98a5e0ee-trusted-ca-bundle\") pod \"ee380d52-b71c-4839-8835-078d98a5e0ee\" (UID: \"ee380d52-b71c-4839-8835-078d98a5e0ee\") "
Oct 30 20:27:41.068704 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:41.068632    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee380d52-b71c-4839-8835-078d98a5e0ee-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ee380d52-b71c-4839-8835-078d98a5e0ee-archives-path\") pod \"ee380d52-b71c-4839-8835-078d98a5e0ee\" (UID: \"ee380d52-b71c-4839-8835-078d98a5e0ee\") "
Oct 30 20:27:41.069006 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:41.068983    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee380d52-b71c-4839-8835-078d98a5e0ee-archives-path" pod "ee380d52-b71c-4839-8835-078d98a5e0ee" (UID: "ee380d52-b71c-4839-8835-078d98a5e0ee"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:27:41.069006 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:41.068990    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee380d52-b71c-4839-8835-078d98a5e0ee-trusted-ca-bundle" pod "ee380d52-b71c-4839-8835-078d98a5e0ee" (UID: "ee380d52-b71c-4839-8835-078d98a5e0ee"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:27:41.070989 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:41.070966    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee380d52-b71c-4839-8835-078d98a5e0ee-kube-api-access-9l2sr" pod "ee380d52-b71c-4839-8835-078d98a5e0ee" (UID: "ee380d52-b71c-4839-8835-078d98a5e0ee"). InnerVolumeSpecName "kube-api-access-9l2sr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:27:41.169214 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:41.169182    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee380d52-b71c-4839-8835-078d98a5e0ee-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee380d52-b71c-4839-8835-078d98a5e0ee-service-ca-bundle\") pod \"ee380d52-b71c-4839-8835-078d98a5e0ee\" (UID: \"ee380d52-b71c-4839-8835-078d98a5e0ee\") "
Oct 30 20:27:41.169373 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:41.169348    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l2sr\" (UniqueName: \"kubernetes.io/projected/ee380d52-b71c-4839-8835-078d98a5e0ee-kube-api-access-9l2sr\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:41.169373 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:41.169359    5071 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee380d52-b71c-4839-8835-078d98a5e0ee-trusted-ca-bundle\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:41.169373 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:41.169370    5071 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ee380d52-b71c-4839-8835-078d98a5e0ee-archives-path\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:41.169501 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:41.169478    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee380d52-b71c-4839-8835-078d98a5e0ee-service-ca-bundle" pod "ee380d52-b71c-4839-8835-078d98a5e0ee" (UID: "ee380d52-b71c-4839-8835-078d98a5e0ee"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:27:41.270562 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:41.270475    5071 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee380d52-b71c-4839-8835-078d98a5e0ee-service-ca-bundle\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:41.672717 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:41.672678    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tp7sc" event={"ID":"5040db9b-91d3-4d89-adaa-d1394cc3a617","Type":"ContainerStarted","Data":"a0359b72aad338c1b358ad3ddb167ced9f043ae69c82ba2ecc24f215af2e1e63"}
Oct 30 20:27:41.673861 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:41.673837    5071 generic.go:356] "Generic (PLEG): container finished" podID="ee380d52-b71c-4839-8835-078d98a5e0ee" containerID="d39f3158be11fb9a929222afc0f19562ec8f1dc76da6a005cce4cb2cb62eeb39" exitCode=2
Oct 30 20:27:41.673980 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:41.673873    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-zmknv-22np8"
Oct 30 20:27:41.673980 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:41.673905    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-zmknv-22np8" event={"ID":"ee380d52-b71c-4839-8835-078d98a5e0ee","Type":"ContainerDied","Data":"d39f3158be11fb9a929222afc0f19562ec8f1dc76da6a005cce4cb2cb62eeb39"}
Oct 30 20:27:41.673980 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:41.673946    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-zmknv-22np8" event={"ID":"ee380d52-b71c-4839-8835-078d98a5e0ee","Type":"ContainerDied","Data":"3ce759cd9d7ce2099096ca884b37156c6e61e84eae795065d82f7c8b5d1a3d18"}
Oct 30 20:27:41.673980 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:41.673964    5071 scope.go:122] "RemoveContainer" containerID="d39f3158be11fb9a929222afc0f19562ec8f1dc76da6a005cce4cb2cb62eeb39"
Oct 30 20:27:41.683152 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:41.683133    5071 scope.go:122] "RemoveContainer" containerID="d39f3158be11fb9a929222afc0f19562ec8f1dc76da6a005cce4cb2cb62eeb39"
Oct 30 20:27:41.683544 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:27:41.683489    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d39f3158be11fb9a929222afc0f19562ec8f1dc76da6a005cce4cb2cb62eeb39\": container with ID starting with d39f3158be11fb9a929222afc0f19562ec8f1dc76da6a005cce4cb2cb62eeb39 not found: ID does not exist" containerID="d39f3158be11fb9a929222afc0f19562ec8f1dc76da6a005cce4cb2cb62eeb39"
Oct 30 20:27:41.683544 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:41.683520    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d39f3158be11fb9a929222afc0f19562ec8f1dc76da6a005cce4cb2cb62eeb39"} err="failed to get container status \"d39f3158be11fb9a929222afc0f19562ec8f1dc76da6a005cce4cb2cb62eeb39\": rpc error: code = NotFound desc = could not find container \"d39f3158be11fb9a929222afc0f19562ec8f1dc76da6a005cce4cb2cb62eeb39\": container with ID starting with d39f3158be11fb9a929222afc0f19562ec8f1dc76da6a005cce4cb2cb62eeb39 not found: ID does not exist"
Oct 30 20:27:41.901444 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:41.901415    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-zmknv-22np8"]
Oct 30 20:27:41.907381 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:41.907359    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-zmknv-22np8"]
Oct 30 20:27:42.199409 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:42.199374    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee380d52-b71c-4839-8835-078d98a5e0ee" path="/var/lib/kubelet/pods/ee380d52-b71c-4839-8835-078d98a5e0ee/volumes"
Oct 30 20:27:44.067974 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:44.067939    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:44.277325 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:44.277294    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:46.726401 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:27:46.726369    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:27:54.497477 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:54.497430    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:27:54.647489 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:27:54.647446    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:28:16.726490 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:28:16.726451    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:28:18.774327 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:28:18.774279    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:28:19.904737 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:28:19.904702    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:28:28.765211 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:28:28.765175    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:28:28.973701 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:28:28.973665    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:28:46.725203 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:28:46.725163    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:29:16.726393 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:29:16.726358    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:29:19.107700 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:29:19.107666    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:29:20.241419 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:29:20.241384    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:29:46.726114 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:29:46.726075    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:29:55.610074 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:29:55.610033    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:29:55.763303 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:29:55.763272    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:02.480758 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:02.480726    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:02.705221 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:02.705180    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:04.041051 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:04.041014    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:04.211978 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:04.211948    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:05.140491 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:05.140452    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:11.289762 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:11.289682    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vhgq7"]
Oct 30 20:30:11.290159 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:11.289997    5071 cpu_manager.go:414] "containerMap: removing container" podUID="05f22644-0884-434d-b56b-193d9554b505" containerName="registry"
Oct 30 20:30:11.290159 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:11.290009    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05f22644-0884-434d-b56b-193d9554b505" containerName="registry"
Oct 30 20:30:11.290159 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:11.290027    5071 cpu_manager.go:414] "containerMap: removing container" podUID="9f908129-4500-4433-871c-032c29a55dd9" containerName="registry"
Oct 30 20:30:11.290159 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:11.290032    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f908129-4500-4433-871c-032c29a55dd9" containerName="registry"
Oct 30 20:30:11.290159 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:11.290039    5071 cpu_manager.go:414] "containerMap: removing container" podUID="ee380d52-b71c-4839-8835-078d98a5e0ee" containerName="insights-gathering"
Oct 30 20:30:11.290159 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:11.290044    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee380d52-b71c-4839-8835-078d98a5e0ee" containerName="insights-gathering"
Oct 30 20:30:11.290159 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:11.290100    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="05f22644-0884-434d-b56b-193d9554b505" containerName="registry"
Oct 30 20:30:11.290159 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:11.290109    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f908129-4500-4433-871c-032c29a55dd9" containerName="registry"
Oct 30 20:30:11.290159 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:11.290114    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee380d52-b71c-4839-8835-078d98a5e0ee" containerName="insights-gathering"
Oct 30 20:30:11.292065 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:11.292049    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vhgq7"
Oct 30 20:30:11.394433 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:11.394398    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 20:30:11.394433 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:11.394400    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 20:30:11.394663 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:11.394568    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-mvprc\""
Oct 30 20:30:11.395665 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:11.395648    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vhgq7"]
Oct 30 20:30:11.425429 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:11.425397    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3cc2515-236f-449a-a0d2-aa476750cf53-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vhgq7\" (UID: \"a3cc2515-236f-449a-a0d2-aa476750cf53\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vhgq7"
Oct 30 20:30:11.425595 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:11.425438    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb4c5\" (UniqueName: \"kubernetes.io/projected/a3cc2515-236f-449a-a0d2-aa476750cf53-kube-api-access-mb4c5\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vhgq7\" (UID: \"a3cc2515-236f-449a-a0d2-aa476750cf53\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vhgq7"
Oct 30 20:30:11.425595 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:11.425463    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3cc2515-236f-449a-a0d2-aa476750cf53-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vhgq7\" (UID: \"a3cc2515-236f-449a-a0d2-aa476750cf53\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vhgq7"
Oct 30 20:30:11.526391 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:11.526360    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3cc2515-236f-449a-a0d2-aa476750cf53-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vhgq7\" (UID: \"a3cc2515-236f-449a-a0d2-aa476750cf53\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vhgq7"
Oct 30 20:30:11.526546 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:11.526401    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb4c5\" (UniqueName: \"kubernetes.io/projected/a3cc2515-236f-449a-a0d2-aa476750cf53-kube-api-access-mb4c5\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vhgq7\" (UID: \"a3cc2515-236f-449a-a0d2-aa476750cf53\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vhgq7"
Oct 30 20:30:11.526546 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:11.526427    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3cc2515-236f-449a-a0d2-aa476750cf53-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vhgq7\" (UID: \"a3cc2515-236f-449a-a0d2-aa476750cf53\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vhgq7"
Oct 30 20:30:11.526757 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:11.526737    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3cc2515-236f-449a-a0d2-aa476750cf53-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vhgq7\" (UID: \"a3cc2515-236f-449a-a0d2-aa476750cf53\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vhgq7"
Oct 30 20:30:11.526801 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:11.526788    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3cc2515-236f-449a-a0d2-aa476750cf53-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vhgq7\" (UID: \"a3cc2515-236f-449a-a0d2-aa476750cf53\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vhgq7"
Oct 30 20:30:11.584649 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:11.584589    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb4c5\" (UniqueName: \"kubernetes.io/projected/a3cc2515-236f-449a-a0d2-aa476750cf53-kube-api-access-mb4c5\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vhgq7\" (UID: \"a3cc2515-236f-449a-a0d2-aa476750cf53\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vhgq7"
Oct 30 20:30:11.604461 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:11.604435    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vhgq7"
Oct 30 20:30:11.796979 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:11.796946    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:11.979389 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:11.979316    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vhgq7"]
Oct 30 20:30:12.006491 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:30:12.006462    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3cc2515_236f_449a_a0d2_aa476750cf53.slice/crio-74ec85599f06081282b3431b869d1f09df70e30cf733f1b812a1efda91fb5c59 WatchSource:0}: Error finding container 74ec85599f06081282b3431b869d1f09df70e30cf733f1b812a1efda91fb5c59: Status 404 returned error can't find the container with id 74ec85599f06081282b3431b869d1f09df70e30cf733f1b812a1efda91fb5c59
Oct 30 20:30:12.049090 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:12.049054    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:12.273563 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:12.273475    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vhgq7" event={"ID":"a3cc2515-236f-449a-a0d2-aa476750cf53","Type":"ContainerStarted","Data":"74ec85599f06081282b3431b869d1f09df70e30cf733f1b812a1efda91fb5c59"}
Oct 30 20:30:13.018021 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.017987    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"]
Oct 30 20:30:13.020514 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.020493    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:30:13.076413 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.076385    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 20:30:13.100315 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.100284    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-hsn45\""
Oct 30 20:30:13.126179 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.126148    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"]
Oct 30 20:30:13.141405 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.141312    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9lhz7\" (UID: \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:30:13.141405 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.141368    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9lhz7\" (UID: \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:30:13.141628 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.141448    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9lhz7\" (UID: \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:30:13.141628 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.141534    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9lhz7\" (UID: \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:30:13.141628 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.141596    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9lhz7\" (UID: \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:30:13.141764 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.141636    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkp5w\" (UniqueName: \"kubernetes.io/projected/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-kube-api-access-nkp5w\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9lhz7\" (UID: \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:30:13.141764 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.141638    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:13.141764 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.141739    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9lhz7\" (UID: \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:30:13.141937 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.141779    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9lhz7\" (UID: \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:30:13.141937 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.141814    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9lhz7\" (UID: \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:30:13.242615 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.242578    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9lhz7\" (UID: \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:30:13.242800 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.242629    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkp5w\" (UniqueName: \"kubernetes.io/projected/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-kube-api-access-nkp5w\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9lhz7\" (UID: \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:30:13.242800 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.242687    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9lhz7\" (UID: \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:30:13.242800 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.242729    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9lhz7\" (UID: \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:30:13.242800 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.242774    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9lhz7\" (UID: \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:30:13.243048 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.242814    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9lhz7\" (UID: \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:30:13.243048 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.242890    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9lhz7\" (UID: \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:30:13.243048 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.242917    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9lhz7\" (UID: \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:30:13.243048 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.242953    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9lhz7\" (UID: \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:30:13.243262 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.243143    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9lhz7\" (UID: \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:30:13.243262 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.243188    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9lhz7\" (UID: \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:30:13.243262 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.243251    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9lhz7\" (UID: \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:30:13.243576 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.243554    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9lhz7\" (UID: \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:30:13.243691 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.243654    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9lhz7\" (UID: \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:30:13.245764 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.245743    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9lhz7\" (UID: \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:30:13.246473 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.246452    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9lhz7\" (UID: \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:30:13.302369 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.302338    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkp5w\" (UniqueName: \"kubernetes.io/projected/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-kube-api-access-nkp5w\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9lhz7\" (UID: \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:30:13.302557 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.302534    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-9lhz7\" (UID: \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:30:13.335214 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.335186    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:30:13.347088 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.347057    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj"]
Oct 30 20:30:13.349616 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.349594    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj"
Oct 30 20:30:13.406625 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.406571    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-dmn9n\""
Oct 30 20:30:13.412065 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.412039    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj"]
Oct 30 20:30:13.445079 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.445043    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/223f5d8a-238d-49f4-9951-0a6a2019cee7-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj\" (UID: \"223f5d8a-238d-49f4-9951-0a6a2019cee7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj"
Oct 30 20:30:13.445222 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.445101    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/223f5d8a-238d-49f4-9951-0a6a2019cee7-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj\" (UID: \"223f5d8a-238d-49f4-9951-0a6a2019cee7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj"
Oct 30 20:30:13.445222 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.445130    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/223f5d8a-238d-49f4-9951-0a6a2019cee7-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj\" (UID: \"223f5d8a-238d-49f4-9951-0a6a2019cee7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj"
Oct 30 20:30:13.445222 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.445176    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/223f5d8a-238d-49f4-9951-0a6a2019cee7-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj\" (UID: \"223f5d8a-238d-49f4-9951-0a6a2019cee7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj"
Oct 30 20:30:13.445222 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.445197    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/223f5d8a-238d-49f4-9951-0a6a2019cee7-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj\" (UID: \"223f5d8a-238d-49f4-9951-0a6a2019cee7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj"
Oct 30 20:30:13.445222 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.445214    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/223f5d8a-238d-49f4-9951-0a6a2019cee7-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj\" (UID: \"223f5d8a-238d-49f4-9951-0a6a2019cee7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj"
Oct 30 20:30:13.445382 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.445229    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/223f5d8a-238d-49f4-9951-0a6a2019cee7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj\" (UID: \"223f5d8a-238d-49f4-9951-0a6a2019cee7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj"
Oct 30 20:30:13.445382 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.445244    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n957t\" (UniqueName: \"kubernetes.io/projected/223f5d8a-238d-49f4-9951-0a6a2019cee7-kube-api-access-n957t\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj\" (UID: \"223f5d8a-238d-49f4-9951-0a6a2019cee7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj"
Oct 30 20:30:13.445382 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.445274    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/223f5d8a-238d-49f4-9951-0a6a2019cee7-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj\" (UID: \"223f5d8a-238d-49f4-9951-0a6a2019cee7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj"
Oct 30 20:30:13.473302 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.473092    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:13.546113 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.546078    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/223f5d8a-238d-49f4-9951-0a6a2019cee7-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj\" (UID: \"223f5d8a-238d-49f4-9951-0a6a2019cee7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj"
Oct 30 20:30:13.546274 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.546133    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/223f5d8a-238d-49f4-9951-0a6a2019cee7-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj\" (UID: \"223f5d8a-238d-49f4-9951-0a6a2019cee7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj"
Oct 30 20:30:13.546274 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.546194    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/223f5d8a-238d-49f4-9951-0a6a2019cee7-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj\" (UID: \"223f5d8a-238d-49f4-9951-0a6a2019cee7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj"
Oct 30 20:30:13.546274 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.546215    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/223f5d8a-238d-49f4-9951-0a6a2019cee7-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj\" (UID: \"223f5d8a-238d-49f4-9951-0a6a2019cee7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj"
Oct 30 20:30:13.546274 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.546232    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/223f5d8a-238d-49f4-9951-0a6a2019cee7-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj\" (UID: \"223f5d8a-238d-49f4-9951-0a6a2019cee7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj"
Oct 30 20:30:13.546274 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.546254    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/223f5d8a-238d-49f4-9951-0a6a2019cee7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj\" (UID: \"223f5d8a-238d-49f4-9951-0a6a2019cee7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj"
Oct 30 20:30:13.546274 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.546271    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n957t\" (UniqueName: \"kubernetes.io/projected/223f5d8a-238d-49f4-9951-0a6a2019cee7-kube-api-access-n957t\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj\" (UID: \"223f5d8a-238d-49f4-9951-0a6a2019cee7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj"
Oct 30 20:30:13.546498 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.546309    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/223f5d8a-238d-49f4-9951-0a6a2019cee7-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj\" (UID: \"223f5d8a-238d-49f4-9951-0a6a2019cee7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj"
Oct 30 20:30:13.546498 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.546337    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/223f5d8a-238d-49f4-9951-0a6a2019cee7-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj\" (UID: \"223f5d8a-238d-49f4-9951-0a6a2019cee7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj"
Oct 30 20:30:13.546631 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.546611    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/223f5d8a-238d-49f4-9951-0a6a2019cee7-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj\" (UID: \"223f5d8a-238d-49f4-9951-0a6a2019cee7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj"
Oct 30 20:30:13.547547 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.546940    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/223f5d8a-238d-49f4-9951-0a6a2019cee7-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj\" (UID: \"223f5d8a-238d-49f4-9951-0a6a2019cee7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj"
Oct 30 20:30:13.547547 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.547201    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/223f5d8a-238d-49f4-9951-0a6a2019cee7-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj\" (UID: \"223f5d8a-238d-49f4-9951-0a6a2019cee7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj"
Oct 30 20:30:13.547547 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.547238    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/223f5d8a-238d-49f4-9951-0a6a2019cee7-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj\" (UID: \"223f5d8a-238d-49f4-9951-0a6a2019cee7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj"
Oct 30 20:30:13.547547 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.547497    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/223f5d8a-238d-49f4-9951-0a6a2019cee7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj\" (UID: \"223f5d8a-238d-49f4-9951-0a6a2019cee7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj"
Oct 30 20:30:13.549628 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.549594    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/223f5d8a-238d-49f4-9951-0a6a2019cee7-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj\" (UID: \"223f5d8a-238d-49f4-9951-0a6a2019cee7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj"
Oct 30 20:30:13.550043 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.550020    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/223f5d8a-238d-49f4-9951-0a6a2019cee7-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj\" (UID: \"223f5d8a-238d-49f4-9951-0a6a2019cee7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj"
Oct 30 20:30:13.605420 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.605350    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/223f5d8a-238d-49f4-9951-0a6a2019cee7-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj\" (UID: \"223f5d8a-238d-49f4-9951-0a6a2019cee7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj"
Oct 30 20:30:13.605420 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.605363    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n957t\" (UniqueName: \"kubernetes.io/projected/223f5d8a-238d-49f4-9951-0a6a2019cee7-kube-api-access-n957t\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj\" (UID: \"223f5d8a-238d-49f4-9951-0a6a2019cee7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj"
Oct 30 20:30:13.665779 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.665740    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj"
Oct 30 20:30:13.688182 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.688149    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:13.767423 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:13.767394    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"]
Oct 30 20:30:13.774886 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:30:13.774857    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1326fe90_f3ab_4922_8ab9_a6eab1dcf665.slice/crio-3950094a7119ca1da065f69c71d7131d56772cd5dfda9bd2000b26cee5990ec9 WatchSource:0}: Error finding container 3950094a7119ca1da065f69c71d7131d56772cd5dfda9bd2000b26cee5990ec9: Status 404 returned error can't find the container with id 3950094a7119ca1da065f69c71d7131d56772cd5dfda9bd2000b26cee5990ec9
Oct 30 20:30:14.047847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:14.047796    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj"]
Oct 30 20:30:14.050588 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:30:14.050555    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod223f5d8a_238d_49f4_9951_0a6a2019cee7.slice/crio-9de27ba81ef1630abf20c67873ca85d94f1c50492a9902d74151f8da720f3d0d WatchSource:0}: Error finding container 9de27ba81ef1630abf20c67873ca85d94f1c50492a9902d74151f8da720f3d0d: Status 404 returned error can't find the container with id 9de27ba81ef1630abf20c67873ca85d94f1c50492a9902d74151f8da720f3d0d
Oct 30 20:30:14.290055 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:14.290013    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj" event={"ID":"223f5d8a-238d-49f4-9951-0a6a2019cee7","Type":"ContainerStarted","Data":"9de27ba81ef1630abf20c67873ca85d94f1c50492a9902d74151f8da720f3d0d"}
Oct 30 20:30:14.291406 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:14.291379    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7" event={"ID":"1326fe90-f3ab-4922-8ab9-a6eab1dcf665","Type":"ContainerStarted","Data":"3950094a7119ca1da065f69c71d7131d56772cd5dfda9bd2000b26cee5990ec9"}
Oct 30 20:30:16.734259 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:30:16.734219    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:30:17.517096 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:17.516968    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:18.573840 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:18.573775    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:30:18.574248 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:18.573894    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:30:18.574248 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:18.573947    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:30:18.621676 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:18.621645    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:30:18.621763 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:18.621716    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:30:18.621763 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:18.621743    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:30:19.318344 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:19.318305    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj" event={"ID":"223f5d8a-238d-49f4-9951-0a6a2019cee7","Type":"ContainerStarted","Data":"57ff8e58118de23cb32822a210c900c87ce270faefd8272a9800abda78c34635"}
Oct 30 20:30:19.319888 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:19.319859    5071 generic.go:356] "Generic (PLEG): container finished" podID="a3cc2515-236f-449a-a0d2-aa476750cf53" containerID="dea4b86e77ac8e360c5586f9ace0d5bc98279e8bd58464c4eea3aea89df156f0" exitCode=0
Oct 30 20:30:19.320029 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:19.319939    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vhgq7" event={"ID":"a3cc2515-236f-449a-a0d2-aa476750cf53","Type":"ContainerDied","Data":"dea4b86e77ac8e360c5586f9ace0d5bc98279e8bd58464c4eea3aea89df156f0"}
Oct 30 20:30:19.321532 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:19.321505    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7" event={"ID":"1326fe90-f3ab-4922-8ab9-a6eab1dcf665","Type":"ContainerStarted","Data":"6f4b4ac41dc3c538c0bc81f7f4833c35ab5de48c80a76db4e1ed45a70038a4a1"}
Oct 30 20:30:19.336005 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:19.335984    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:30:19.340628 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:19.340610    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:30:19.470074 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:19.470037    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:19.486025 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:19.485974    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj" podStartSLOduration=1.966094101 podStartE2EDuration="6.485961046s" totalImagesPullingTime="4.519866945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:19.485153735 +0000 UTC m=+231.402175347" watchObservedRunningTime="2026-10-30 20:30:19.485961046 +0000 UTC m=+231.402982582"
Oct 30 20:30:19.666324 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:19.666242    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj"
Oct 30 20:30:19.670721 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:19.670700    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj"
Oct 30 20:30:19.726033 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:19.725987    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7" podStartSLOduration=2.882489172 podStartE2EDuration="7.725971662s" totalImagesPullingTime="4.84348249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:19.72512407 +0000 UTC m=+231.642145606" watchObservedRunningTime="2026-10-30 20:30:19.725971662 +0000 UTC m=+231.642993198"
Oct 30 20:30:20.326627 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:20.326587    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:30:20.326848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:20.326663    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj"
Oct 30 20:30:20.327752 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:20.327727    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj"
Oct 30 20:30:20.327890 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:20.327785    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:30:20.573874 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:20.573815    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:22.609375 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:22.609292    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:23.348080 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:23.348045    5071 generic.go:356] "Generic (PLEG): container finished" podID="a3cc2515-236f-449a-a0d2-aa476750cf53" containerID="460803457cf60a7e07b4ae96a2631b3a5e31f0b2f6c9134a90497f8b353ffcdc" exitCode=0
Oct 30 20:30:23.348304 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:23.348123    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vhgq7" event={"ID":"a3cc2515-236f-449a-a0d2-aa476750cf53","Type":"ContainerDied","Data":"460803457cf60a7e07b4ae96a2631b3a5e31f0b2f6c9134a90497f8b353ffcdc"}
Oct 30 20:30:30.384159 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:30.384121    5071 generic.go:356] "Generic (PLEG): container finished" podID="a3cc2515-236f-449a-a0d2-aa476750cf53" containerID="c9acdb86f8727c31ba52ed601560e7cb1546815e25b1bb46617ce93d79fd45ec" exitCode=0
Oct 30 20:30:30.384582 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:30.384213    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vhgq7" event={"ID":"a3cc2515-236f-449a-a0d2-aa476750cf53","Type":"ContainerDied","Data":"c9acdb86f8727c31ba52ed601560e7cb1546815e25b1bb46617ce93d79fd45ec"}
Oct 30 20:30:31.518960 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:31.518937    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vhgq7"
Oct 30 20:30:31.605212 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:31.605182    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3cc2515-236f-449a-a0d2-aa476750cf53-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3cc2515-236f-449a-a0d2-aa476750cf53-bundle\") pod \"a3cc2515-236f-449a-a0d2-aa476750cf53\" (UID: \"a3cc2515-236f-449a-a0d2-aa476750cf53\") "
Oct 30 20:30:31.605326 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:31.605244    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3cc2515-236f-449a-a0d2-aa476750cf53-kube-api-access-mb4c5\" (UniqueName: \"kubernetes.io/projected/a3cc2515-236f-449a-a0d2-aa476750cf53-kube-api-access-mb4c5\") pod \"a3cc2515-236f-449a-a0d2-aa476750cf53\" (UID: \"a3cc2515-236f-449a-a0d2-aa476750cf53\") "
Oct 30 20:30:31.605326 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:31.605291    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3cc2515-236f-449a-a0d2-aa476750cf53-util\" (UniqueName: \"kubernetes.io/empty-dir/a3cc2515-236f-449a-a0d2-aa476750cf53-util\") pod \"a3cc2515-236f-449a-a0d2-aa476750cf53\" (UID: \"a3cc2515-236f-449a-a0d2-aa476750cf53\") "
Oct 30 20:30:31.605627 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:31.605599    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3cc2515-236f-449a-a0d2-aa476750cf53-bundle" pod "a3cc2515-236f-449a-a0d2-aa476750cf53" (UID: "a3cc2515-236f-449a-a0d2-aa476750cf53"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:31.607618 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:31.607595    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3cc2515-236f-449a-a0d2-aa476750cf53-kube-api-access-mb4c5" pod "a3cc2515-236f-449a-a0d2-aa476750cf53" (UID: "a3cc2515-236f-449a-a0d2-aa476750cf53"). InnerVolumeSpecName "kube-api-access-mb4c5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:31.609295 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:31.609269    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3cc2515-236f-449a-a0d2-aa476750cf53-util" pod "a3cc2515-236f-449a-a0d2-aa476750cf53" (UID: "a3cc2515-236f-449a-a0d2-aa476750cf53"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:31.706261 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:31.706185    5071 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3cc2515-236f-449a-a0d2-aa476750cf53-bundle\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:31.706261 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:31.706213    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb4c5\" (UniqueName: \"kubernetes.io/projected/a3cc2515-236f-449a-a0d2-aa476750cf53-kube-api-access-mb4c5\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:31.706261 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:31.706224    5071 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3cc2515-236f-449a-a0d2-aa476750cf53-util\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:32.393997 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:32.393955    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vhgq7" event={"ID":"a3cc2515-236f-449a-a0d2-aa476750cf53","Type":"ContainerDied","Data":"74ec85599f06081282b3431b869d1f09df70e30cf733f1b812a1efda91fb5c59"}
Oct 30 20:30:32.393997 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:32.393983    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vhgq7"
Oct 30 20:30:32.393997 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:32.394001    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74ec85599f06081282b3431b869d1f09df70e30cf733f1b812a1efda91fb5c59"
Oct 30 20:30:36.642924 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:36.642885    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:38.038801 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:38.038762    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:42.233813 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:42.233779    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:42.481125 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:42.481089    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:43.452042 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.452008    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:43.601911 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.601878    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-dz2mf"]
Oct 30 20:30:43.602208 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.602197    5071 cpu_manager.go:414] "containerMap: removing container" podUID="a3cc2515-236f-449a-a0d2-aa476750cf53" containerName="extract"
Oct 30 20:30:43.602251 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.602210    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3cc2515-236f-449a-a0d2-aa476750cf53" containerName="extract"
Oct 30 20:30:43.602251 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.602224    5071 cpu_manager.go:414] "containerMap: removing container" podUID="a3cc2515-236f-449a-a0d2-aa476750cf53" containerName="util"
Oct 30 20:30:43.602251 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.602230    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3cc2515-236f-449a-a0d2-aa476750cf53" containerName="util"
Oct 30 20:30:43.602251 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.602241    5071 cpu_manager.go:414] "containerMap: removing container" podUID="a3cc2515-236f-449a-a0d2-aa476750cf53" containerName="pull"
Oct 30 20:30:43.602251 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.602246    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3cc2515-236f-449a-a0d2-aa476750cf53" containerName="pull"
Oct 30 20:30:43.602402 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.602298    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3cc2515-236f-449a-a0d2-aa476750cf53" containerName="extract"
Oct 30 20:30:43.604298 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.604283    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-dz2mf"
Oct 30 20:30:43.657980 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.657947    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 20:30:43.658158 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.658049    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 20:30:43.695839 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.695794    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:43.705880 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.705784    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/494fdf1e-6527-4270-8f76-530e17a6c900-webhook-cert\") pod \"opendatahub-operator-controller-manager-fdf79688b-dz2mf\" (UID: \"494fdf1e-6527-4270-8f76-530e17a6c900\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-dz2mf"
Oct 30 20:30:43.705880 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.705842    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/494fdf1e-6527-4270-8f76-530e17a6c900-tmp\") pod \"opendatahub-operator-controller-manager-fdf79688b-dz2mf\" (UID: \"494fdf1e-6527-4270-8f76-530e17a6c900\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-dz2mf"
Oct 30 20:30:43.706017 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.705943    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/494fdf1e-6527-4270-8f76-530e17a6c900-apiservice-cert\") pod \"opendatahub-operator-controller-manager-fdf79688b-dz2mf\" (UID: \"494fdf1e-6527-4270-8f76-530e17a6c900\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-dz2mf"
Oct 30 20:30:43.706017 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.705998    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/494fdf1e-6527-4270-8f76-530e17a6c900-metrics-cert\") pod \"opendatahub-operator-controller-manager-fdf79688b-dz2mf\" (UID: \"494fdf1e-6527-4270-8f76-530e17a6c900\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-dz2mf"
Oct 30 20:30:43.706083 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.706023    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/494fdf1e-6527-4270-8f76-530e17a6c900-manifests\") pod \"opendatahub-operator-controller-manager-fdf79688b-dz2mf\" (UID: \"494fdf1e-6527-4270-8f76-530e17a6c900\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-dz2mf"
Oct 30 20:30:43.706158 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.706140    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pwxl\" (UniqueName: \"kubernetes.io/projected/494fdf1e-6527-4270-8f76-530e17a6c900-kube-api-access-7pwxl\") pod \"opendatahub-operator-controller-manager-fdf79688b-dz2mf\" (UID: \"494fdf1e-6527-4270-8f76-530e17a6c900\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-dz2mf"
Oct 30 20:30:43.707000 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.706984    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 20:30:43.707080 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.706998    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 20:30:43.707080 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.707014    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 20:30:43.707145 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.707104    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-jjpq2\""
Oct 30 20:30:43.717415 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.717390    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-dz2mf"]
Oct 30 20:30:43.806968 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.806936    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/494fdf1e-6527-4270-8f76-530e17a6c900-webhook-cert\") pod \"opendatahub-operator-controller-manager-fdf79688b-dz2mf\" (UID: \"494fdf1e-6527-4270-8f76-530e17a6c900\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-dz2mf"
Oct 30 20:30:43.807147 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.806974    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/494fdf1e-6527-4270-8f76-530e17a6c900-tmp\") pod \"opendatahub-operator-controller-manager-fdf79688b-dz2mf\" (UID: \"494fdf1e-6527-4270-8f76-530e17a6c900\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-dz2mf"
Oct 30 20:30:43.807147 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.806995    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/494fdf1e-6527-4270-8f76-530e17a6c900-apiservice-cert\") pod \"opendatahub-operator-controller-manager-fdf79688b-dz2mf\" (UID: \"494fdf1e-6527-4270-8f76-530e17a6c900\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-dz2mf"
Oct 30 20:30:43.807147 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.807019    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/494fdf1e-6527-4270-8f76-530e17a6c900-metrics-cert\") pod \"opendatahub-operator-controller-manager-fdf79688b-dz2mf\" (UID: \"494fdf1e-6527-4270-8f76-530e17a6c900\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-dz2mf"
Oct 30 20:30:43.807147 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.807048    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/494fdf1e-6527-4270-8f76-530e17a6c900-manifests\") pod \"opendatahub-operator-controller-manager-fdf79688b-dz2mf\" (UID: \"494fdf1e-6527-4270-8f76-530e17a6c900\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-dz2mf"
Oct 30 20:30:43.807147 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.807117    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pwxl\" (UniqueName: \"kubernetes.io/projected/494fdf1e-6527-4270-8f76-530e17a6c900-kube-api-access-7pwxl\") pod \"opendatahub-operator-controller-manager-fdf79688b-dz2mf\" (UID: \"494fdf1e-6527-4270-8f76-530e17a6c900\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-dz2mf"
Oct 30 20:30:43.807556 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.807515    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/494fdf1e-6527-4270-8f76-530e17a6c900-manifests\") pod \"opendatahub-operator-controller-manager-fdf79688b-dz2mf\" (UID: \"494fdf1e-6527-4270-8f76-530e17a6c900\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-dz2mf"
Oct 30 20:30:43.809551 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.809525    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/494fdf1e-6527-4270-8f76-530e17a6c900-tmp\") pod \"opendatahub-operator-controller-manager-fdf79688b-dz2mf\" (UID: \"494fdf1e-6527-4270-8f76-530e17a6c900\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-dz2mf"
Oct 30 20:30:43.809959 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.809934    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/494fdf1e-6527-4270-8f76-530e17a6c900-webhook-cert\") pod \"opendatahub-operator-controller-manager-fdf79688b-dz2mf\" (UID: \"494fdf1e-6527-4270-8f76-530e17a6c900\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-dz2mf"
Oct 30 20:30:43.809959 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.809951    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/494fdf1e-6527-4270-8f76-530e17a6c900-apiservice-cert\") pod \"opendatahub-operator-controller-manager-fdf79688b-dz2mf\" (UID: \"494fdf1e-6527-4270-8f76-530e17a6c900\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-dz2mf"
Oct 30 20:30:43.810164 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.810149    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/494fdf1e-6527-4270-8f76-530e17a6c900-metrics-cert\") pod \"opendatahub-operator-controller-manager-fdf79688b-dz2mf\" (UID: \"494fdf1e-6527-4270-8f76-530e17a6c900\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-dz2mf"
Oct 30 20:30:43.866343 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.866314    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pwxl\" (UniqueName: \"kubernetes.io/projected/494fdf1e-6527-4270-8f76-530e17a6c900-kube-api-access-7pwxl\") pod \"opendatahub-operator-controller-manager-fdf79688b-dz2mf\" (UID: \"494fdf1e-6527-4270-8f76-530e17a6c900\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-dz2mf"
Oct 30 20:30:43.873883 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.873856    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:43.916925 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:43.916899    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-dz2mf"
Oct 30 20:30:44.046184 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:44.046150    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:44.395644 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:44.395622    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-dz2mf"]
Oct 30 20:30:44.397460 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:30:44.397427    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod494fdf1e_6527_4270_8f76_530e17a6c900.slice/crio-b647efc6a21b61cf2f84d5092f4bbb8ae765c78e02d9026f4af844f874afac0f WatchSource:0}: Error finding container b647efc6a21b61cf2f84d5092f4bbb8ae765c78e02d9026f4af844f874afac0f: Status 404 returned error can't find the container with id b647efc6a21b61cf2f84d5092f4bbb8ae765c78e02d9026f4af844f874afac0f
Oct 30 20:30:44.442716 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:44.442672    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-dz2mf" event={"ID":"494fdf1e-6527-4270-8f76-530e17a6c900","Type":"ContainerStarted","Data":"b647efc6a21b61cf2f84d5092f4bbb8ae765c78e02d9026f4af844f874afac0f"}
Oct 30 20:30:46.241685 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:46.241645    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:46.728445 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:30:46.728405    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:30:46.958056 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:46.958024    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:47.128155 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:47.128121    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:48.463976 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:48.463941    5071 generic.go:356] "Generic (PLEG): container finished" podID="494fdf1e-6527-4270-8f76-530e17a6c900" containerID="22e66aaab2affced8a17435e9148c23d2598375f116d6d175b6ea7a6b80a2f22" exitCode=0
Oct 30 20:30:48.464383 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:48.464006    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-dz2mf" event={"ID":"494fdf1e-6527-4270-8f76-530e17a6c900","Type":"ContainerDied","Data":"22e66aaab2affced8a17435e9148c23d2598375f116d6d175b6ea7a6b80a2f22"}
Oct 30 20:30:49.470078 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:49.470045    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-dz2mf" event={"ID":"494fdf1e-6527-4270-8f76-530e17a6c900","Type":"ContainerStarted","Data":"a4f998428b48853543a4a64b093fddb822d29af03e25e1f78c495936857eb9d7"}
Oct 30 20:30:49.470522 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:49.470126    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-dz2mf"
Oct 30 20:30:49.700121 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:49.700067    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-dz2mf" podStartSLOduration=2.030941424 podStartE2EDuration="6.700052704s" totalImagesPullingTime="4.218347749s" totalInitContainerRuntime="450.763531ms" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:49.649498884 +0000 UTC m=+261.566520421" watchObservedRunningTime="2026-10-30 20:30:49.700052704 +0000 UTC m=+261.617074310"
Oct 30 20:30:49.942246 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:49.942198    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:51.141259 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:51.141216    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:51.936209 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:51.936167    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:52.982916 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:52.982872    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:53.205241 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:53.205202    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:57.142280 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:57.142242    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:30:58.306075 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:58.306035    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rl87m"]
Oct 30 20:30:58.339608 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:58.339577    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rl87m"
Oct 30 20:30:58.364400 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:58.364370    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rl87m"]
Oct 30 20:30:58.397656 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:58.397612    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 20:30:58.412252 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:58.412210    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 20:30:58.412464 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:58.412448    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-f5v2p\""
Oct 30 20:30:58.434890 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:58.434861    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cptrn\" (UniqueName: \"kubernetes.io/projected/c941da87-4602-489f-b174-19d18a687d61-kube-api-access-cptrn\") pod \"kuadrant-operator-controller-manager-cf8469d7b-rl87m\" (UID: \"c941da87-4602-489f-b174-19d18a687d61\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rl87m"
Oct 30 20:30:58.435026 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:58.434973    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c941da87-4602-489f-b174-19d18a687d61-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-rl87m\" (UID: \"c941da87-4602-489f-b174-19d18a687d61\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rl87m"
Oct 30 20:30:58.535575 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:58.535547    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cptrn\" (UniqueName: \"kubernetes.io/projected/c941da87-4602-489f-b174-19d18a687d61-kube-api-access-cptrn\") pod \"kuadrant-operator-controller-manager-cf8469d7b-rl87m\" (UID: \"c941da87-4602-489f-b174-19d18a687d61\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rl87m"
Oct 30 20:30:58.535778 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:58.535607    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c941da87-4602-489f-b174-19d18a687d61-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-rl87m\" (UID: \"c941da87-4602-489f-b174-19d18a687d61\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rl87m"
Oct 30 20:30:58.536047 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:58.536026    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c941da87-4602-489f-b174-19d18a687d61-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-rl87m\" (UID: \"c941da87-4602-489f-b174-19d18a687d61\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rl87m"
Oct 30 20:30:58.601752 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:58.601689    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cptrn\" (UniqueName: \"kubernetes.io/projected/c941da87-4602-489f-b174-19d18a687d61-kube-api-access-cptrn\") pod \"kuadrant-operator-controller-manager-cf8469d7b-rl87m\" (UID: \"c941da87-4602-489f-b174-19d18a687d61\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rl87m"
Oct 30 20:30:58.666021 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:58.665985    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rl87m"
Oct 30 20:30:59.160391 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:59.160363    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rl87m"]
Oct 30 20:30:59.198903 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:30:59.198873    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc941da87_4602_489f_b174_19d18a687d61.slice/crio-6e17f4df389b61fc19db311d7b753a35fc5417acef5d3be17f85a14378338697 WatchSource:0}: Error finding container 6e17f4df389b61fc19db311d7b753a35fc5417acef5d3be17f85a14378338697: Status 404 returned error can't find the container with id 6e17f4df389b61fc19db311d7b753a35fc5417acef5d3be17f85a14378338697
Oct 30 20:30:59.511541 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:59.511464    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rl87m" event={"ID":"c941da87-4602-489f-b174-19d18a687d61","Type":"ContainerStarted","Data":"6e17f4df389b61fc19db311d7b753a35fc5417acef5d3be17f85a14378338697"}
Oct 30 20:30:59.540128 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:30:59.540086    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:00.476227 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:00.476201    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fdf79688b-dz2mf"
Oct 30 20:31:01.044245 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:01.044209    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jsbv8"]
Oct 30 20:31:01.079191 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:01.079163    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jsbv8"
Oct 30 20:31:01.111733 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:01.111706    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jsbv8"]
Oct 30 20:31:01.133070 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:01.133039    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:31:01.133210 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:01.133123    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 30 20:31:01.133210 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:01.133138    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-vhjnc\""
Oct 30 20:31:01.261599 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:01.261566    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txcwr\" (UniqueName: \"kubernetes.io/projected/6e4f602e-2e7c-4838-ae19-fc20ddc1879f-kube-api-access-txcwr\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-jsbv8\" (UID: \"6e4f602e-2e7c-4838-ae19-fc20ddc1879f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jsbv8"
Oct 30 20:31:01.261599 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:01.261604    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e4f602e-2e7c-4838-ae19-fc20ddc1879f-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-jsbv8\" (UID: \"6e4f602e-2e7c-4838-ae19-fc20ddc1879f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jsbv8"
Oct 30 20:31:01.355390 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:01.355267    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:01.362375 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:01.362347    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txcwr\" (UniqueName: \"kubernetes.io/projected/6e4f602e-2e7c-4838-ae19-fc20ddc1879f-kube-api-access-txcwr\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-jsbv8\" (UID: \"6e4f602e-2e7c-4838-ae19-fc20ddc1879f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jsbv8"
Oct 30 20:31:01.362528 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:01.362385    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e4f602e-2e7c-4838-ae19-fc20ddc1879f-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-jsbv8\" (UID: \"6e4f602e-2e7c-4838-ae19-fc20ddc1879f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jsbv8"
Oct 30 20:31:01.362876 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:01.362856    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e4f602e-2e7c-4838-ae19-fc20ddc1879f-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-jsbv8\" (UID: \"6e4f602e-2e7c-4838-ae19-fc20ddc1879f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jsbv8"
Oct 30 20:31:01.422932 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:01.422890    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txcwr\" (UniqueName: \"kubernetes.io/projected/6e4f602e-2e7c-4838-ae19-fc20ddc1879f-kube-api-access-txcwr\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-jsbv8\" (UID: \"6e4f602e-2e7c-4838-ae19-fc20ddc1879f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jsbv8"
Oct 30 20:31:01.704285 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:01.704191    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jsbv8"
Oct 30 20:31:02.180170 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:02.180147    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jsbv8"]
Oct 30 20:31:02.183028 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:31:02.182996    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e4f602e_2e7c_4838_ae19_fc20ddc1879f.slice/crio-76628b895988530169b94528bde4a19e90b952838731afebdcf923f7a89b3581 WatchSource:0}: Error finding container 76628b895988530169b94528bde4a19e90b952838731afebdcf923f7a89b3581: Status 404 returned error can't find the container with id 76628b895988530169b94528bde4a19e90b952838731afebdcf923f7a89b3581
Oct 30 20:31:02.526995 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:02.526911    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jsbv8" event={"ID":"6e4f602e-2e7c-4838-ae19-fc20ddc1879f","Type":"ContainerStarted","Data":"76628b895988530169b94528bde4a19e90b952838731afebdcf923f7a89b3581"}
Oct 30 20:31:03.201854 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:03.201800    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:03.517943 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:03.517800    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:03.702345 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:03.702316    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:03.880295 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:03.880211    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:06.046409 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:06.046369    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-tvzgz"]
Oct 30 20:31:06.054528 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:06.054501    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tvzgz"
Oct 30 20:31:06.114170 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:06.114139    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 30 20:31:06.160813 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:06.160762    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-nlfnb\""
Oct 30 20:31:06.162073 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:06.162041    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-tvzgz"]
Oct 30 20:31:06.209391 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:06.209359    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x4k8\" (UniqueName: \"kubernetes.io/projected/e9bd0a5f-c80e-416d-a40c-6fd2def09183-kube-api-access-8x4k8\") pod \"dns-operator-controller-manager-8557cbdb84-tvzgz\" (UID: \"e9bd0a5f-c80e-416d-a40c-6fd2def09183\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tvzgz"
Oct 30 20:31:06.310653 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:06.310588    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x4k8\" (UniqueName: \"kubernetes.io/projected/e9bd0a5f-c80e-416d-a40c-6fd2def09183-kube-api-access-8x4k8\") pod \"dns-operator-controller-manager-8557cbdb84-tvzgz\" (UID: \"e9bd0a5f-c80e-416d-a40c-6fd2def09183\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tvzgz"
Oct 30 20:31:06.354858 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:06.354772    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:06.372457 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:06.372426    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x4k8\" (UniqueName: \"kubernetes.io/projected/e9bd0a5f-c80e-416d-a40c-6fd2def09183-kube-api-access-8x4k8\") pod \"dns-operator-controller-manager-8557cbdb84-tvzgz\" (UID: \"e9bd0a5f-c80e-416d-a40c-6fd2def09183\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tvzgz"
Oct 30 20:31:06.549859 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:06.549804    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rl87m" event={"ID":"c941da87-4602-489f-b174-19d18a687d61","Type":"ContainerStarted","Data":"83a63ec3356d8ebb5f187db042d436760bf74cf4a58c6b6d52492fab18cf7935"}
Oct 30 20:31:06.550033 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:06.549971    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rl87m"
Oct 30 20:31:06.670475 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:06.670438    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tvzgz"
Oct 30 20:31:06.675859 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:06.675072    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rl87m" podStartSLOduration=1.661483778 podStartE2EDuration="8.675052609s" totalImagesPullingTime="7.013568831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:06.673343926 +0000 UTC m=+278.590365470" watchObservedRunningTime="2026-10-30 20:31:06.675052609 +0000 UTC m=+278.592074148"
Oct 30 20:31:06.977984 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:06.977793    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:07.068019 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:07.067994    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-tvzgz"]
Oct 30 20:31:07.070694 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:31:07.070661    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9bd0a5f_c80e_416d_a40c_6fd2def09183.slice/crio-2b83a6c1745ba3cda4dd621330e131ab286807fddd25fcfbbad4011b2f705834 WatchSource:0}: Error finding container 2b83a6c1745ba3cda4dd621330e131ab286807fddd25fcfbbad4011b2f705834: Status 404 returned error can't find the container with id 2b83a6c1745ba3cda4dd621330e131ab286807fddd25fcfbbad4011b2f705834
Oct 30 20:31:07.249280 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:07.249205    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:07.556367 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:07.556330    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jsbv8" event={"ID":"6e4f602e-2e7c-4838-ae19-fc20ddc1879f","Type":"ContainerStarted","Data":"1afc6af109a792646926c88e703f6da0784e28b7847ee00856ef59f44de7a300"}
Oct 30 20:31:07.557771 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:07.557745    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tvzgz" event={"ID":"e9bd0a5f-c80e-416d-a40c-6fd2def09183","Type":"ContainerStarted","Data":"2b83a6c1745ba3cda4dd621330e131ab286807fddd25fcfbbad4011b2f705834"}
Oct 30 20:31:07.688698 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:07.688647    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jsbv8" podStartSLOduration=2.297910242 podStartE2EDuration="6.688627057s" totalImagesPullingTime="4.390716815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:07.686560491 +0000 UTC m=+279.603582027" watchObservedRunningTime="2026-10-30 20:31:07.688627057 +0000 UTC m=+279.605648594"
Oct 30 20:31:08.603845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:08.603786    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:08.862336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:08.862238    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:09.162052 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:09.161956    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:10.552121 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:10.552084    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:10.993532 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:10.993462    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-4j4v2"]
Oct 30 20:31:10.995786 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:10.995771    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-4j4v2"
Oct 30 20:31:11.051281 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:11.051250    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l9qc\" (UniqueName: \"kubernetes.io/projected/da53ef69-95be-48e9-971e-4168bf928443-kube-api-access-9l9qc\") pod \"authorino-operator-7b7bb4bc77-4j4v2\" (UID: \"da53ef69-95be-48e9-971e-4168bf928443\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-4j4v2"
Oct 30 20:31:11.101036 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:11.101005    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-2jd5s\""
Oct 30 20:31:11.102722 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:11.102698    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-4j4v2"]
Oct 30 20:31:11.152069 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:11.152039    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l9qc\" (UniqueName: \"kubernetes.io/projected/da53ef69-95be-48e9-971e-4168bf928443-kube-api-access-9l9qc\") pod \"authorino-operator-7b7bb4bc77-4j4v2\" (UID: \"da53ef69-95be-48e9-971e-4168bf928443\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-4j4v2"
Oct 30 20:31:11.211222 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:11.211196    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l9qc\" (UniqueName: \"kubernetes.io/projected/da53ef69-95be-48e9-971e-4168bf928443-kube-api-access-9l9qc\") pod \"authorino-operator-7b7bb4bc77-4j4v2\" (UID: \"da53ef69-95be-48e9-971e-4168bf928443\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-4j4v2"
Oct 30 20:31:11.308131 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:11.308101    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-4j4v2"
Oct 30 20:31:11.340092 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:11.340045    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:11.692655 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:11.692632    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-4j4v2"]
Oct 30 20:31:11.695102 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:31:11.695076    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda53ef69_95be_48e9_971e_4168bf928443.slice/crio-cc2ed0fe3ec5c7ff1be25e39c807ae137544ee2783f9f9c4d88fb5cbe5935701 WatchSource:0}: Error finding container cc2ed0fe3ec5c7ff1be25e39c807ae137544ee2783f9f9c4d88fb5cbe5935701: Status 404 returned error can't find the container with id cc2ed0fe3ec5c7ff1be25e39c807ae137544ee2783f9f9c4d88fb5cbe5935701
Oct 30 20:31:11.733223 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:11.733188    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:11.972410 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:11.972322    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:12.580721 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:12.580691    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-4j4v2" event={"ID":"da53ef69-95be-48e9-971e-4168bf928443","Type":"ContainerStarted","Data":"cc2ed0fe3ec5c7ff1be25e39c807ae137544ee2783f9f9c4d88fb5cbe5935701"}
Oct 30 20:31:12.613215 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:12.613180    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:12.884247 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:12.884142    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:14.950159 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:14.950120    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:15.162943 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:15.162912    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:15.385131 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:15.385063    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:15.601749 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:15.601711    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:15.602384 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:15.602351    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tvzgz" event={"ID":"e9bd0a5f-c80e-416d-a40c-6fd2def09183","Type":"ContainerStarted","Data":"ff5eb3b7074af1559d0858695739d0fa79c873c0de2ab7b6f69384b79ff824bf"}
Oct 30 20:31:15.727647 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:15.727582    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tvzgz" podStartSLOduration=2.611027429 podStartE2EDuration="10.727564205s" totalImagesPullingTime="8.116536776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:15.726034112 +0000 UTC m=+287.643055649" watchObservedRunningTime="2026-10-30 20:31:15.727564205 +0000 UTC m=+287.644585740"
Oct 30 20:31:15.820037 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:15.820000    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:16.104208 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:16.104169    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:16.343714 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:16.343658    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:16.608067 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:16.608033    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tvzgz"
Oct 30 20:31:16.727165 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:31:16.727134    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:31:17.335373 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:17.335337    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:17.525119 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:17.525090    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:17.560480 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:17.560451    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rl87m"
Oct 30 20:31:17.615010 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:17.614928    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-4j4v2" event={"ID":"da53ef69-95be-48e9-971e-4168bf928443","Type":"ContainerStarted","Data":"4875294955dd77a3acba89f8f2d47f8f2e6d0e9758c51f0c228de17219a5d34b"}
Oct 30 20:31:17.852465 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:17.852413    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-4j4v2" podStartSLOduration=2.8831707399999997 podStartE2EDuration="7.852396934s" totalImagesPullingTime="4.969226194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:17.850548942 +0000 UTC m=+289.767570478" watchObservedRunningTime="2026-10-30 20:31:17.852396934 +0000 UTC m=+289.769418470"
Oct 30 20:31:18.109346 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.109308    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2"]
Oct 30 20:31:18.117107 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.117088    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2"
Oct 30 20:31:18.175667 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.175634    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2"]
Oct 30 20:31:18.185177 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.185145    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:18.197242 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.197219    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-4m77g\""
Oct 30 20:31:18.217489 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.217459    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97751fc9-3d1c-451d-b2e3-7fbc522d3ad9-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2\" (UID: \"97751fc9-3d1c-451d-b2e3-7fbc522d3ad9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2"
Oct 30 20:31:18.217651 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.217525    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97751fc9-3d1c-451d-b2e3-7fbc522d3ad9-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2\" (UID: \"97751fc9-3d1c-451d-b2e3-7fbc522d3ad9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2"
Oct 30 20:31:18.217651 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.217579    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97751fc9-3d1c-451d-b2e3-7fbc522d3ad9-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2\" (UID: \"97751fc9-3d1c-451d-b2e3-7fbc522d3ad9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2"
Oct 30 20:31:18.217651 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.217599    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfgsd\" (UniqueName: \"kubernetes.io/projected/97751fc9-3d1c-451d-b2e3-7fbc522d3ad9-kube-api-access-zfgsd\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2\" (UID: \"97751fc9-3d1c-451d-b2e3-7fbc522d3ad9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2"
Oct 30 20:31:18.217803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.217686    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97751fc9-3d1c-451d-b2e3-7fbc522d3ad9-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2\" (UID: \"97751fc9-3d1c-451d-b2e3-7fbc522d3ad9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2"
Oct 30 20:31:18.217803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.217725    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97751fc9-3d1c-451d-b2e3-7fbc522d3ad9-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2\" (UID: \"97751fc9-3d1c-451d-b2e3-7fbc522d3ad9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2"
Oct 30 20:31:18.217934 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.217849    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97751fc9-3d1c-451d-b2e3-7fbc522d3ad9-istio-data\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2\" (UID: \"97751fc9-3d1c-451d-b2e3-7fbc522d3ad9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2"
Oct 30 20:31:18.217996 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.217928    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97751fc9-3d1c-451d-b2e3-7fbc522d3ad9-istio-token\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2\" (UID: \"97751fc9-3d1c-451d-b2e3-7fbc522d3ad9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2"
Oct 30 20:31:18.217996 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.217960    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97751fc9-3d1c-451d-b2e3-7fbc522d3ad9-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2\" (UID: \"97751fc9-3d1c-451d-b2e3-7fbc522d3ad9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2"
Oct 30 20:31:18.318703 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.318658    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfgsd\" (UniqueName: \"kubernetes.io/projected/97751fc9-3d1c-451d-b2e3-7fbc522d3ad9-kube-api-access-zfgsd\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2\" (UID: \"97751fc9-3d1c-451d-b2e3-7fbc522d3ad9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2"
Oct 30 20:31:18.318937 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.318708    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97751fc9-3d1c-451d-b2e3-7fbc522d3ad9-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2\" (UID: \"97751fc9-3d1c-451d-b2e3-7fbc522d3ad9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2"
Oct 30 20:31:18.318937 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.318737    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97751fc9-3d1c-451d-b2e3-7fbc522d3ad9-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2\" (UID: \"97751fc9-3d1c-451d-b2e3-7fbc522d3ad9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2"
Oct 30 20:31:18.318937 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.318760    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97751fc9-3d1c-451d-b2e3-7fbc522d3ad9-istio-data\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2\" (UID: \"97751fc9-3d1c-451d-b2e3-7fbc522d3ad9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2"
Oct 30 20:31:18.318937 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.318796    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97751fc9-3d1c-451d-b2e3-7fbc522d3ad9-istio-token\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2\" (UID: \"97751fc9-3d1c-451d-b2e3-7fbc522d3ad9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2"
Oct 30 20:31:18.319120 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.318931    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97751fc9-3d1c-451d-b2e3-7fbc522d3ad9-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2\" (UID: \"97751fc9-3d1c-451d-b2e3-7fbc522d3ad9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2"
Oct 30 20:31:18.319120 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.319062    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97751fc9-3d1c-451d-b2e3-7fbc522d3ad9-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2\" (UID: \"97751fc9-3d1c-451d-b2e3-7fbc522d3ad9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2"
Oct 30 20:31:18.319120 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.319100    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97751fc9-3d1c-451d-b2e3-7fbc522d3ad9-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2\" (UID: \"97751fc9-3d1c-451d-b2e3-7fbc522d3ad9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2"
Oct 30 20:31:18.319276 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.319132    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97751fc9-3d1c-451d-b2e3-7fbc522d3ad9-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2\" (UID: \"97751fc9-3d1c-451d-b2e3-7fbc522d3ad9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2"
Oct 30 20:31:18.319276 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.319158    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97751fc9-3d1c-451d-b2e3-7fbc522d3ad9-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2\" (UID: \"97751fc9-3d1c-451d-b2e3-7fbc522d3ad9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2"
Oct 30 20:31:18.319276 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.319209    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97751fc9-3d1c-451d-b2e3-7fbc522d3ad9-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2\" (UID: \"97751fc9-3d1c-451d-b2e3-7fbc522d3ad9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2"
Oct 30 20:31:18.319276 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.319260    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97751fc9-3d1c-451d-b2e3-7fbc522d3ad9-istio-data\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2\" (UID: \"97751fc9-3d1c-451d-b2e3-7fbc522d3ad9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2"
Oct 30 20:31:18.319630 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.319600    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97751fc9-3d1c-451d-b2e3-7fbc522d3ad9-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2\" (UID: \"97751fc9-3d1c-451d-b2e3-7fbc522d3ad9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2"
Oct 30 20:31:18.320009 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.319989    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97751fc9-3d1c-451d-b2e3-7fbc522d3ad9-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2\" (UID: \"97751fc9-3d1c-451d-b2e3-7fbc522d3ad9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2"
Oct 30 20:31:18.321480 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.321459    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97751fc9-3d1c-451d-b2e3-7fbc522d3ad9-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2\" (UID: \"97751fc9-3d1c-451d-b2e3-7fbc522d3ad9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2"
Oct 30 20:31:18.321687 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.321669    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97751fc9-3d1c-451d-b2e3-7fbc522d3ad9-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2\" (UID: \"97751fc9-3d1c-451d-b2e3-7fbc522d3ad9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2"
Oct 30 20:31:18.384700 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.384615    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:18.386915 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.386887    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfgsd\" (UniqueName: \"kubernetes.io/projected/97751fc9-3d1c-451d-b2e3-7fbc522d3ad9-kube-api-access-zfgsd\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2\" (UID: \"97751fc9-3d1c-451d-b2e3-7fbc522d3ad9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2"
Oct 30 20:31:18.388108 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.388079    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97751fc9-3d1c-451d-b2e3-7fbc522d3ad9-istio-token\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2\" (UID: \"97751fc9-3d1c-451d-b2e3-7fbc522d3ad9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2"
Oct 30 20:31:18.430100 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.430071    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2"
Oct 30 20:31:18.440581 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.440558    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9zb25"]
Oct 30 20:31:18.444312 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.444298    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9zb25"
Oct 30 20:31:18.507920 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.507892    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9zb25"]
Oct 30 20:31:18.520691 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.520665    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6mg9\" (UniqueName: \"kubernetes.io/projected/934f75e2-5d3c-4df4-88ea-57007d36d328-kube-api-access-k6mg9\") pod \"kuadrant-operator-controller-manager-678d87766f-9zb25\" (UID: \"934f75e2-5d3c-4df4-88ea-57007d36d328\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9zb25"
Oct 30 20:31:18.520840 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.520699    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/934f75e2-5d3c-4df4-88ea-57007d36d328-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-9zb25\" (UID: \"934f75e2-5d3c-4df4-88ea-57007d36d328\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9zb25"
Oct 30 20:31:18.585123 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.585075    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:18.618990 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.618956    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-4j4v2"
Oct 30 20:31:18.621464 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.621437    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6mg9\" (UniqueName: \"kubernetes.io/projected/934f75e2-5d3c-4df4-88ea-57007d36d328-kube-api-access-k6mg9\") pod \"kuadrant-operator-controller-manager-678d87766f-9zb25\" (UID: \"934f75e2-5d3c-4df4-88ea-57007d36d328\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9zb25"
Oct 30 20:31:18.621464 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.621467    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/934f75e2-5d3c-4df4-88ea-57007d36d328-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-9zb25\" (UID: \"934f75e2-5d3c-4df4-88ea-57007d36d328\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9zb25"
Oct 30 20:31:18.621800 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.621783    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/934f75e2-5d3c-4df4-88ea-57007d36d328-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-9zb25\" (UID: \"934f75e2-5d3c-4df4-88ea-57007d36d328\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9zb25"
Oct 30 20:31:18.680695 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.680620    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6mg9\" (UniqueName: \"kubernetes.io/projected/934f75e2-5d3c-4df4-88ea-57007d36d328-kube-api-access-k6mg9\") pod \"kuadrant-operator-controller-manager-678d87766f-9zb25\" (UID: \"934f75e2-5d3c-4df4-88ea-57007d36d328\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9zb25"
Oct 30 20:31:18.763744 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.763712    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9zb25"
Oct 30 20:31:18.860257 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.860232    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2"]
Oct 30 20:31:18.861772 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:31:18.861744    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97751fc9_3d1c_451d_b2e3_7fbc522d3ad9.slice/crio-534c12dce8890c145af7bf8cf3f5dc072b780382d3e57287cb053415a1ae7782 WatchSource:0}: Error finding container 534c12dce8890c145af7bf8cf3f5dc072b780382d3e57287cb053415a1ae7782: Status 404 returned error can't find the container with id 534c12dce8890c145af7bf8cf3f5dc072b780382d3e57287cb053415a1ae7782
Oct 30 20:31:18.863959 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.863924    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:31:18.864049 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.863988    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:31:18.864049 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.864023    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:31:18.955066 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:18.954954    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:19.140470 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:19.140353    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9zb25"]
Oct 30 20:31:19.143696 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:31:19.143668    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod934f75e2_5d3c_4df4_88ea_57007d36d328.slice/crio-77dc715e683a5ff4897c8694fbd81244b69d32af8f99b5db5c1f49c99879ffc4 WatchSource:0}: Error finding container 77dc715e683a5ff4897c8694fbd81244b69d32af8f99b5db5c1f49c99879ffc4: Status 404 returned error can't find the container with id 77dc715e683a5ff4897c8694fbd81244b69d32af8f99b5db5c1f49c99879ffc4
Oct 30 20:31:19.629366 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:19.629328    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9zb25" event={"ID":"934f75e2-5d3c-4df4-88ea-57007d36d328","Type":"ContainerStarted","Data":"86b0d7361c1fef6a086447a5a86d69241b8b5258df80034087865320adc0812c"}
Oct 30 20:31:19.629901 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:19.629426    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9zb25" event={"ID":"934f75e2-5d3c-4df4-88ea-57007d36d328","Type":"ContainerStarted","Data":"77dc715e683a5ff4897c8694fbd81244b69d32af8f99b5db5c1f49c99879ffc4"}
Oct 30 20:31:19.629901 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:19.629641    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9zb25"
Oct 30 20:31:19.631719 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:19.631692    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2" event={"ID":"97751fc9-3d1c-451d-b2e3-7fbc522d3ad9","Type":"ContainerStarted","Data":"247137c62606ed333e3e11a6674c9f63b18e6225f3ab68a4cac2fbd56da9862e"}
Oct 30 20:31:19.632369 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:19.631742    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2" event={"ID":"97751fc9-3d1c-451d-b2e3-7fbc522d3ad9","Type":"ContainerStarted","Data":"534c12dce8890c145af7bf8cf3f5dc072b780382d3e57287cb053415a1ae7782"}
Oct 30 20:31:19.803179 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:19.803146    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:19.803732 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:19.803690    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9zb25" podStartSLOduration=1.8036650349999999 podStartE2EDuration="1.803665035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:19.802178978 +0000 UTC m=+291.719200517" watchObservedRunningTime="2026-10-30 20:31:19.803665035 +0000 UTC m=+291.720686572"
Oct 30 20:31:19.921900 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:19.921767    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2" podStartSLOduration=1.9217514690000002 podStartE2EDuration="1.921751469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:19.921263713 +0000 UTC m=+291.838285288" watchObservedRunningTime="2026-10-30 20:31:19.921751469 +0000 UTC m=+291.838773004"
Oct 30 20:31:20.011764 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:20.011726    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:20.430748 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:20.430714    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2"
Oct 30 20:31:20.435576 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:20.435550    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2"
Oct 30 20:31:20.637027 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:20.636993    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2"
Oct 30 20:31:20.637978 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:20.637959    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2"
Oct 30 20:31:20.919609 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:20.919567    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:21.549895 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:21.549861    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:21.803907 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:21.803800    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:22.941324 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:22.941293    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:23.152585 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:23.152547    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:23.355891 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:23.355854    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:25.658554 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:25.658511    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:26.148802 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:26.148765    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:26.312804 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:26.312770    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:27.617438 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:27.617409    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tvzgz"
Oct 30 20:31:28.103449 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:28.103406    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:28.223850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:28.223798    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8qvb_91fe9e06-0c30-4000-bddb-2ba89cd048cc/console-operator/2.log"
Oct 30 20:31:28.224244 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:28.224228    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8qvb_91fe9e06-0c30-4000-bddb-2ba89cd048cc/console-operator/2.log"
Oct 30 20:31:28.231736 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:28.231714    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q4f4h_80cae79e-b60f-4efe-bb58-e77753a457e5/ovn-acl-logging/0.log"
Oct 30 20:31:28.231895 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:28.231777    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q4f4h_80cae79e-b60f-4efe-bb58-e77753a457e5/ovn-acl-logging/0.log"
Oct 30 20:31:29.634921 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:29.634885    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-4j4v2"
Oct 30 20:31:30.639266 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:30.639232    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9zb25"
Oct 30 20:31:30.882145 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:30.882114    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rl87m"]
Oct 30 20:31:30.882370 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:30.882346    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rl87m" podUID="c941da87-4602-489f-b174-19d18a687d61" containerName="manager" containerID="cri-o://83a63ec3356d8ebb5f187db042d436760bf74cf4a58c6b6d52492fab18cf7935" gracePeriod=10
Oct 30 20:31:31.129022 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:31.128999    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rl87m"
Oct 30 20:31:31.233370 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:31.233290    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c941da87-4602-489f-b174-19d18a687d61-kube-api-access-cptrn\" (UniqueName: \"kubernetes.io/projected/c941da87-4602-489f-b174-19d18a687d61-kube-api-access-cptrn\") pod \"c941da87-4602-489f-b174-19d18a687d61\" (UID: \"c941da87-4602-489f-b174-19d18a687d61\") "
Oct 30 20:31:31.233370 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:31.233356    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c941da87-4602-489f-b174-19d18a687d61-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c941da87-4602-489f-b174-19d18a687d61-extensions-socket-volume\") pod \"c941da87-4602-489f-b174-19d18a687d61\" (UID: \"c941da87-4602-489f-b174-19d18a687d61\") "
Oct 30 20:31:31.233788 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:31.233758    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c941da87-4602-489f-b174-19d18a687d61-extensions-socket-volume" pod "c941da87-4602-489f-b174-19d18a687d61" (UID: "c941da87-4602-489f-b174-19d18a687d61"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:31.236190 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:31.236160    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c941da87-4602-489f-b174-19d18a687d61-kube-api-access-cptrn" pod "c941da87-4602-489f-b174-19d18a687d61" (UID: "c941da87-4602-489f-b174-19d18a687d61"). InnerVolumeSpecName "kube-api-access-cptrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:31.334880 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:31.334845    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cptrn\" (UniqueName: \"kubernetes.io/projected/c941da87-4602-489f-b174-19d18a687d61-kube-api-access-cptrn\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:31.334880 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:31.334873    5071 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c941da87-4602-489f-b174-19d18a687d61-extensions-socket-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:31.689348 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:31.689313    5071 generic.go:356] "Generic (PLEG): container finished" podID="c941da87-4602-489f-b174-19d18a687d61" containerID="83a63ec3356d8ebb5f187db042d436760bf74cf4a58c6b6d52492fab18cf7935" exitCode=0
Oct 30 20:31:31.689793 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:31.689366    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rl87m"
Oct 30 20:31:31.689793 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:31.689393    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rl87m" event={"ID":"c941da87-4602-489f-b174-19d18a687d61","Type":"ContainerDied","Data":"83a63ec3356d8ebb5f187db042d436760bf74cf4a58c6b6d52492fab18cf7935"}
Oct 30 20:31:31.689793 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:31.689447    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rl87m" event={"ID":"c941da87-4602-489f-b174-19d18a687d61","Type":"ContainerDied","Data":"6e17f4df389b61fc19db311d7b753a35fc5417acef5d3be17f85a14378338697"}
Oct 30 20:31:31.689793 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:31.689470    5071 scope.go:122] "RemoveContainer" containerID="83a63ec3356d8ebb5f187db042d436760bf74cf4a58c6b6d52492fab18cf7935"
Oct 30 20:31:31.694839 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:31.694805    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:31.698709 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:31.698690    5071 scope.go:122] "RemoveContainer" containerID="83a63ec3356d8ebb5f187db042d436760bf74cf4a58c6b6d52492fab18cf7935"
Oct 30 20:31:31.699011 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:31:31.698993    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83a63ec3356d8ebb5f187db042d436760bf74cf4a58c6b6d52492fab18cf7935\": container with ID starting with 83a63ec3356d8ebb5f187db042d436760bf74cf4a58c6b6d52492fab18cf7935 not found: ID does not exist" containerID="83a63ec3356d8ebb5f187db042d436760bf74cf4a58c6b6d52492fab18cf7935"
Oct 30 20:31:31.699081 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:31.699019    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83a63ec3356d8ebb5f187db042d436760bf74cf4a58c6b6d52492fab18cf7935"} err="failed to get container status \"83a63ec3356d8ebb5f187db042d436760bf74cf4a58c6b6d52492fab18cf7935\": rpc error: code = NotFound desc = could not find container \"83a63ec3356d8ebb5f187db042d436760bf74cf4a58c6b6d52492fab18cf7935\": container with ID starting with 83a63ec3356d8ebb5f187db042d436760bf74cf4a58c6b6d52492fab18cf7935 not found: ID does not exist"
Oct 30 20:31:31.879971 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:31.879929    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rl87m"]
Oct 30 20:31:31.929782 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:31.929754    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rl87m"]
Oct 30 20:31:31.965466 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:31.965381    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:32.199894 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:32.199859    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c941da87-4602-489f-b174-19d18a687d61" path="/var/lib/kubelet/pods/c941da87-4602-489f-b174-19d18a687d61/volumes"
Oct 30 20:31:36.461401 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:36.461349    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:36.683991 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:36.683953    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:41.941706 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:41.941620    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:42.106131 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:42.106098    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:43.478810 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:43.478773    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:43.638225 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:43.638191    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:45.929843 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:45.929788    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:46.096540 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:46.096509    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:46.727393 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:31:46.727354    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:31:47.699136 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:47.699100    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-lkmv2"]
Oct 30 20:31:47.699527 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:47.699512    5071 cpu_manager.go:414] "containerMap: removing container" podUID="c941da87-4602-489f-b174-19d18a687d61" containerName="manager"
Oct 30 20:31:47.699571 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:47.699527    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c941da87-4602-489f-b174-19d18a687d61" containerName="manager"
Oct 30 20:31:47.699621 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:47.699610    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="c941da87-4602-489f-b174-19d18a687d61" containerName="manager"
Oct 30 20:31:47.704486 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:47.704463    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lkmv2"
Oct 30 20:31:47.758231 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:47.758205    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-bzhzj\""
Oct 30 20:31:47.777681 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:47.777657    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2064d6a5-f7fb-46fa-b655-7366285e6148-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-lkmv2\" (UID: \"2064d6a5-f7fb-46fa-b655-7366285e6148\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lkmv2"
Oct 30 20:31:47.777857 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:47.777688    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2064d6a5-f7fb-46fa-b655-7366285e6148-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-lkmv2\" (UID: \"2064d6a5-f7fb-46fa-b655-7366285e6148\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lkmv2"
Oct 30 20:31:47.777857 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:47.777708    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2064d6a5-f7fb-46fa-b655-7366285e6148-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-lkmv2\" (UID: \"2064d6a5-f7fb-46fa-b655-7366285e6148\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lkmv2"
Oct 30 20:31:47.777857 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:47.777730    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbpxh\" (UniqueName: \"kubernetes.io/projected/2064d6a5-f7fb-46fa-b655-7366285e6148-kube-api-access-wbpxh\") pod \"kserve-module-controller-manager-7566cf5fc8-lkmv2\" (UID: \"2064d6a5-f7fb-46fa-b655-7366285e6148\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lkmv2"
Oct 30 20:31:47.806968 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:47.806944    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 20:31:47.807115 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:47.807042    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 20:31:47.878488 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:47.878454    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2064d6a5-f7fb-46fa-b655-7366285e6148-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-lkmv2\" (UID: \"2064d6a5-f7fb-46fa-b655-7366285e6148\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lkmv2"
Oct 30 20:31:47.878488 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:47.878499    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2064d6a5-f7fb-46fa-b655-7366285e6148-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-lkmv2\" (UID: \"2064d6a5-f7fb-46fa-b655-7366285e6148\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lkmv2"
Oct 30 20:31:47.878756 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:47.878625    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2064d6a5-f7fb-46fa-b655-7366285e6148-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-lkmv2\" (UID: \"2064d6a5-f7fb-46fa-b655-7366285e6148\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lkmv2"
Oct 30 20:31:47.878756 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:47.878657    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbpxh\" (UniqueName: \"kubernetes.io/projected/2064d6a5-f7fb-46fa-b655-7366285e6148-kube-api-access-wbpxh\") pod \"kserve-module-controller-manager-7566cf5fc8-lkmv2\" (UID: \"2064d6a5-f7fb-46fa-b655-7366285e6148\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lkmv2"
Oct 30 20:31:47.878910 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:47.878857    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2064d6a5-f7fb-46fa-b655-7366285e6148-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-lkmv2\" (UID: \"2064d6a5-f7fb-46fa-b655-7366285e6148\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lkmv2"
Oct 30 20:31:47.878968 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:47.878939    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2064d6a5-f7fb-46fa-b655-7366285e6148-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-lkmv2\" (UID: \"2064d6a5-f7fb-46fa-b655-7366285e6148\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lkmv2"
Oct 30 20:31:47.906990 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:47.906958    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 30 20:31:47.909238 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:47.909209    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-lkmv2"]
Oct 30 20:31:47.909987 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:47.909966    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2064d6a5-f7fb-46fa-b655-7366285e6148-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-lkmv2\" (UID: \"2064d6a5-f7fb-46fa-b655-7366285e6148\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lkmv2"
Oct 30 20:31:47.937026 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:47.936993    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbpxh\" (UniqueName: \"kubernetes.io/projected/2064d6a5-f7fb-46fa-b655-7366285e6148-kube-api-access-wbpxh\") pod \"kserve-module-controller-manager-7566cf5fc8-lkmv2\" (UID: \"2064d6a5-f7fb-46fa-b655-7366285e6148\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lkmv2"
Oct 30 20:31:48.017321 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:48.017246    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lkmv2"
Oct 30 20:31:48.439865 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:48.439814    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-lkmv2"]
Oct 30 20:31:48.441650 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:31:48.441624    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2064d6a5_f7fb_46fa_b655_7366285e6148.slice/crio-3a633f99fad10808697a4e6291167bbcd538fe4802c2d14b176b6bba64cdded8 WatchSource:0}: Error finding container 3a633f99fad10808697a4e6291167bbcd538fe4802c2d14b176b6bba64cdded8: Status 404 returned error can't find the container with id 3a633f99fad10808697a4e6291167bbcd538fe4802c2d14b176b6bba64cdded8
Oct 30 20:31:48.443616 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:48.443598    5071 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:31:48.775182 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:48.775092    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lkmv2" event={"ID":"2064d6a5-f7fb-46fa-b655-7366285e6148","Type":"ContainerStarted","Data":"3a633f99fad10808697a4e6291167bbcd538fe4802c2d14b176b6bba64cdded8"}
Oct 30 20:31:49.721140 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:49.721092    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:31:49.724230 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:49.724196    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"
Oct 30 20:31:49.778254 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:49.778223    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 20:31:49.828005 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:49.827972    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6tg6g\""
Oct 30 20:31:49.830443 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:49.830406    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:31:49.895187 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:49.895147    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8ea98f67-8b88-46cb-abac-ed7153834e6a-config-file\") pod \"limitador-limitador-7c9b87d574-pwbqs\" (UID: \"8ea98f67-8b88-46cb-abac-ed7153834e6a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"
Oct 30 20:31:49.895369 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:49.895227    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fvzc\" (UniqueName: \"kubernetes.io/projected/8ea98f67-8b88-46cb-abac-ed7153834e6a-kube-api-access-8fvzc\") pod \"limitador-limitador-7c9b87d574-pwbqs\" (UID: \"8ea98f67-8b88-46cb-abac-ed7153834e6a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"
Oct 30 20:31:49.996651 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:49.996571    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8ea98f67-8b88-46cb-abac-ed7153834e6a-config-file\") pod \"limitador-limitador-7c9b87d574-pwbqs\" (UID: \"8ea98f67-8b88-46cb-abac-ed7153834e6a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"
Oct 30 20:31:49.996940 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:49.996645    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fvzc\" (UniqueName: \"kubernetes.io/projected/8ea98f67-8b88-46cb-abac-ed7153834e6a-kube-api-access-8fvzc\") pod \"limitador-limitador-7c9b87d574-pwbqs\" (UID: \"8ea98f67-8b88-46cb-abac-ed7153834e6a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"
Oct 30 20:31:49.997295 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:49.997268    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8ea98f67-8b88-46cb-abac-ed7153834e6a-config-file\") pod \"limitador-limitador-7c9b87d574-pwbqs\" (UID: \"8ea98f67-8b88-46cb-abac-ed7153834e6a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"
Oct 30 20:31:50.018971 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:50.018940    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:31:50.055541 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:50.055515    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fvzc\" (UniqueName: \"kubernetes.io/projected/8ea98f67-8b88-46cb-abac-ed7153834e6a-kube-api-access-8fvzc\") pod \"limitador-limitador-7c9b87d574-pwbqs\" (UID: \"8ea98f67-8b88-46cb-abac-ed7153834e6a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"
Oct 30 20:31:50.094815 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:50.094758    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:50.297338 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:50.297302    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:50.340107 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:50.340072    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"
Oct 30 20:31:50.721813 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:50.721242    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:31:50.725635 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:31:50.725590    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea98f67_8b88_46cb_abac_ed7153834e6a.slice/crio-e081f440c484278b196bc0e38704045354657cbb778d6111f8ebbd0b74b8614e WatchSource:0}: Error finding container e081f440c484278b196bc0e38704045354657cbb778d6111f8ebbd0b74b8614e: Status 404 returned error can't find the container with id e081f440c484278b196bc0e38704045354657cbb778d6111f8ebbd0b74b8614e
Oct 30 20:31:50.788046 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:50.788010    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-pwbqs" event={"ID":"8ea98f67-8b88-46cb-abac-ed7153834e6a","Type":"ContainerStarted","Data":"e081f440c484278b196bc0e38704045354657cbb778d6111f8ebbd0b74b8614e"}
Oct 30 20:31:51.520314 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:51.520284    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:51.686494 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:51.686408    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:51.794055 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:51.794021    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lkmv2" event={"ID":"2064d6a5-f7fb-46fa-b655-7366285e6148","Type":"ContainerStarted","Data":"a54f990bcb945f002cc60b1cf10ce4da347326568c096a02cb2eacc0e9e5738f"}
Oct 30 20:31:51.794451 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:51.794160    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lkmv2"
Oct 30 20:31:51.961002 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:51.960900    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lkmv2" podStartSLOduration=1.9780490560000001 podStartE2EDuration="4.96088621s" totalImagesPullingTime="2.982837154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:51.959535886 +0000 UTC m=+323.876557422" watchObservedRunningTime="2026-10-30 20:31:51.96088621 +0000 UTC m=+323.877907745"
Oct 30 20:31:52.483170 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:52.483125    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:52.701309 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:52.701275    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:53.813565 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:53.813524    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-pwbqs" event={"ID":"8ea98f67-8b88-46cb-abac-ed7153834e6a","Type":"ContainerStarted","Data":"6ea02d451bc6b6d0299505b787681c283802e15f9e865b3f1b1ead4df799cb69"}
Oct 30 20:31:53.814060 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:53.813615    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"
Oct 30 20:31:53.929959 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:53.929911    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-pwbqs" podStartSLOduration=2.9173850569999997 podStartE2EDuration="4.92989619s" totalImagesPullingTime="2.012511133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:53.927914478 +0000 UTC m=+325.844936011" watchObservedRunningTime="2026-10-30 20:31:53.92989619 +0000 UTC m=+325.846917779"
Oct 30 20:31:57.685962 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:57.685924    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:57.892525 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:57.892489    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:58.086353 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:58.086308    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:58.250875 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:58.250810    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:31:58.418137 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:31:58.418039    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:04.819897 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:04.819865    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"
Oct 30 20:32:04.893135 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:04.893096    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:05.078722 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:05.078620    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:05.437236 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:05.437149    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:05.598664 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:05.598631    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:05.753934 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:05.753856    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:07.371896 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:07.371842    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:07.535009 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:07.534972    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:12.807954 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:12.807924    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lkmv2"
Oct 30 20:32:13.926062 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:13.926024    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:14.141403 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:14.141363    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:14.350019 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:14.349982    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:15.047895 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:15.047861    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:16.726611 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:32:16.726577    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:32:20.144109 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:20.144070    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:21.255050 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:21.255014    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:23.258772 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:23.258735    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:36.638446 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:36.638406    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:36.681797 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:36.681753    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-9pj25"]
Oct 30 20:32:36.687779 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:36.687759    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-9pj25"
Oct 30 20:32:36.744694 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:36.744663    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-9pj25"]
Oct 30 20:32:36.791923 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:36.791884    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 20:32:36.791923 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:36.791887    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 20:32:36.792172 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:36.791940    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 20:32:36.792172 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:36.791997    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sk69l\""
Oct 30 20:32:36.797772 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:36.797742    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:36.819896 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:36.819869    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3730060c-0d34-4d0f-af06-dee152463807-webhook-cert\") pod \"maas-controller-579cf7c6d-9pj25\" (UID: \"3730060c-0d34-4d0f-af06-dee152463807\") " pod="opendatahub/maas-controller-579cf7c6d-9pj25"
Oct 30 20:32:36.820044 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:36.819920    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3730060c-0d34-4d0f-af06-dee152463807-metrics-certs\") pod \"maas-controller-579cf7c6d-9pj25\" (UID: \"3730060c-0d34-4d0f-af06-dee152463807\") " pod="opendatahub/maas-controller-579cf7c6d-9pj25"
Oct 30 20:32:36.820044 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:36.819940    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhn5t\" (UniqueName: \"kubernetes.io/projected/3730060c-0d34-4d0f-af06-dee152463807-kube-api-access-dhn5t\") pod \"maas-controller-579cf7c6d-9pj25\" (UID: \"3730060c-0d34-4d0f-af06-dee152463807\") " pod="opendatahub/maas-controller-579cf7c6d-9pj25"
Oct 30 20:32:36.921235 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:36.921153    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3730060c-0d34-4d0f-af06-dee152463807-webhook-cert\") pod \"maas-controller-579cf7c6d-9pj25\" (UID: \"3730060c-0d34-4d0f-af06-dee152463807\") " pod="opendatahub/maas-controller-579cf7c6d-9pj25"
Oct 30 20:32:36.921235 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:36.921203    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3730060c-0d34-4d0f-af06-dee152463807-metrics-certs\") pod \"maas-controller-579cf7c6d-9pj25\" (UID: \"3730060c-0d34-4d0f-af06-dee152463807\") " pod="opendatahub/maas-controller-579cf7c6d-9pj25"
Oct 30 20:32:36.921463 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:36.921323    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhn5t\" (UniqueName: \"kubernetes.io/projected/3730060c-0d34-4d0f-af06-dee152463807-kube-api-access-dhn5t\") pod \"maas-controller-579cf7c6d-9pj25\" (UID: \"3730060c-0d34-4d0f-af06-dee152463807\") " pod="opendatahub/maas-controller-579cf7c6d-9pj25"
Oct 30 20:32:36.923807 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:36.923781    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3730060c-0d34-4d0f-af06-dee152463807-webhook-cert\") pod \"maas-controller-579cf7c6d-9pj25\" (UID: \"3730060c-0d34-4d0f-af06-dee152463807\") " pod="opendatahub/maas-controller-579cf7c6d-9pj25"
Oct 30 20:32:36.923941 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:36.923854    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3730060c-0d34-4d0f-af06-dee152463807-metrics-certs\") pod \"maas-controller-579cf7c6d-9pj25\" (UID: \"3730060c-0d34-4d0f-af06-dee152463807\") " pod="opendatahub/maas-controller-579cf7c6d-9pj25"
Oct 30 20:32:36.956209 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:36.956174    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:36.980100 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:36.980076    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhn5t\" (UniqueName: \"kubernetes.io/projected/3730060c-0d34-4d0f-af06-dee152463807-kube-api-access-dhn5t\") pod \"maas-controller-579cf7c6d-9pj25\" (UID: \"3730060c-0d34-4d0f-af06-dee152463807\") " pod="opendatahub/maas-controller-579cf7c6d-9pj25"
Oct 30 20:32:36.999789 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:36.999763    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-9pj25"
Oct 30 20:32:37.112433 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:37.112399    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6565584cfb-w8rp4"]
Oct 30 20:32:37.114988 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:37.114971    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6565584cfb-w8rp4"
Oct 30 20:32:37.121787 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:37.121754    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:37.177368 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:37.177287    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6565584cfb-w8rp4"]
Oct 30 20:32:37.223988 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:37.223955    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tsvl\" (UniqueName: \"kubernetes.io/projected/b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607-kube-api-access-5tsvl\") pod \"maas-controller-6565584cfb-w8rp4\" (UID: \"b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607\") " pod="opendatahub/maas-controller-6565584cfb-w8rp4"
Oct 30 20:32:37.224138 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:37.224017    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607-webhook-cert\") pod \"maas-controller-6565584cfb-w8rp4\" (UID: \"b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607\") " pod="opendatahub/maas-controller-6565584cfb-w8rp4"
Oct 30 20:32:37.224184 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:37.224143    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607-metrics-certs\") pod \"maas-controller-6565584cfb-w8rp4\" (UID: \"b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607\") " pod="opendatahub/maas-controller-6565584cfb-w8rp4"
Oct 30 20:32:37.325463 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:37.325425    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607-webhook-cert\") pod \"maas-controller-6565584cfb-w8rp4\" (UID: \"b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607\") " pod="opendatahub/maas-controller-6565584cfb-w8rp4"
Oct 30 20:32:37.325656 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:37.325490    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607-metrics-certs\") pod \"maas-controller-6565584cfb-w8rp4\" (UID: \"b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607\") " pod="opendatahub/maas-controller-6565584cfb-w8rp4"
Oct 30 20:32:37.325656 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:37.325560    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tsvl\" (UniqueName: \"kubernetes.io/projected/b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607-kube-api-access-5tsvl\") pod \"maas-controller-6565584cfb-w8rp4\" (UID: \"b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607\") " pod="opendatahub/maas-controller-6565584cfb-w8rp4"
Oct 30 20:32:37.328107 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:37.328079    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607-metrics-certs\") pod \"maas-controller-6565584cfb-w8rp4\" (UID: \"b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607\") " pod="opendatahub/maas-controller-6565584cfb-w8rp4"
Oct 30 20:32:37.328107 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:37.328091    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607-webhook-cert\") pod \"maas-controller-6565584cfb-w8rp4\" (UID: \"b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607\") " pod="opendatahub/maas-controller-6565584cfb-w8rp4"
Oct 30 20:32:37.371334 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:37.371313    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-9pj25"]
Oct 30 20:32:37.373490 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:32:37.373459    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3730060c_0d34_4d0f_af06_dee152463807.slice/crio-7da6ba73780e9718f6e3a7c032267b227fa4e30cae4970523ce6eba2dd05a0bf WatchSource:0}: Error finding container 7da6ba73780e9718f6e3a7c032267b227fa4e30cae4970523ce6eba2dd05a0bf: Status 404 returned error can't find the container with id 7da6ba73780e9718f6e3a7c032267b227fa4e30cae4970523ce6eba2dd05a0bf
Oct 30 20:32:37.383942 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:37.383921    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tsvl\" (UniqueName: \"kubernetes.io/projected/b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607-kube-api-access-5tsvl\") pod \"maas-controller-6565584cfb-w8rp4\" (UID: \"b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607\") " pod="opendatahub/maas-controller-6565584cfb-w8rp4"
Oct 30 20:32:37.429432 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:37.429346    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6565584cfb-w8rp4"
Oct 30 20:32:37.801680 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:37.801655    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6565584cfb-w8rp4"]
Oct 30 20:32:37.803773 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:32:37.803747    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2ecc2aa_5d8a_400a_b7c4_e4d55be1e607.slice/crio-47dc5b1498c12482d873f500e1201846a0bcaffc1dba81308f8a29369ee86440 WatchSource:0}: Error finding container 47dc5b1498c12482d873f500e1201846a0bcaffc1dba81308f8a29369ee86440: Status 404 returned error can't find the container with id 47dc5b1498c12482d873f500e1201846a0bcaffc1dba81308f8a29369ee86440
Oct 30 20:32:38.028608 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:38.028572    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6565584cfb-w8rp4" event={"ID":"b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607","Type":"ContainerStarted","Data":"47dc5b1498c12482d873f500e1201846a0bcaffc1dba81308f8a29369ee86440"}
Oct 30 20:32:38.029837 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:38.029802    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-9pj25" event={"ID":"3730060c-0d34-4d0f-af06-dee152463807","Type":"ContainerStarted","Data":"7da6ba73780e9718f6e3a7c032267b227fa4e30cae4970523ce6eba2dd05a0bf"}
Oct 30 20:32:41.049771 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:41.049717    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-9pj25" event={"ID":"3730060c-0d34-4d0f-af06-dee152463807","Type":"ContainerStarted","Data":"255e160fa7112afc5cec753793f383db4e6766be8e4d88a102ffe74046295f9a"}
Oct 30 20:32:41.050235 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:41.049878    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-9pj25"
Oct 30 20:32:41.219810 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:41.219615    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-9pj25" podStartSLOduration=1.67884651 podStartE2EDuration="5.219596771s" totalImagesPullingTime="3.540750261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:32:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:32:41.219495598 +0000 UTC m=+373.136517135" watchObservedRunningTime="2026-10-30 20:32:41.219596771 +0000 UTC m=+373.136618308"
Oct 30 20:32:41.406766 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:41.406728    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:41.753872 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:41.753801    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:42.055591 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:42.055548    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6565584cfb-w8rp4" event={"ID":"b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607","Type":"ContainerStarted","Data":"d1edfe6aec2489767e8aa2e1c677191ae5cc7c382ab8e800953737d207cf3e61"}
Oct 30 20:32:42.176310 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:42.176262    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6565584cfb-w8rp4" podStartSLOduration=1.792258419 podStartE2EDuration="5.176249285s" totalImagesPullingTime="3.383990866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:32:42.175639802 +0000 UTC m=+374.092661339" watchObservedRunningTime="2026-10-30 20:32:42.176249285 +0000 UTC m=+374.093270873"
Oct 30 20:32:43.061780 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:43.061750    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6565584cfb-w8rp4"
Oct 30 20:32:44.766573 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:44.766535    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:45.082950 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:45.082917    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:46.727489 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:32:46.727451    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:32:49.779508 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:49.779474    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jscm6"]
Oct 30 20:32:49.786661 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:49.786640    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jscm6"
Oct 30 20:32:49.844299 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:49.844263    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 20:32:49.844484 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:49.844315    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 20:32:49.844484 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:49.844341    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 20:32:49.855011 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:49.854984    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/21b86bc7-8a12-4fdc-9074-b044bbf04f4c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jscm6\" (UID: \"21b86bc7-8a12-4fdc-9074-b044bbf04f4c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jscm6"
Oct 30 20:32:49.855098 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:49.855042    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/21b86bc7-8a12-4fdc-9074-b044bbf04f4c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jscm6\" (UID: \"21b86bc7-8a12-4fdc-9074-b044bbf04f4c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jscm6"
Oct 30 20:32:49.855098 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:49.855086    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxscw\" (UniqueName: \"kubernetes.io/projected/21b86bc7-8a12-4fdc-9074-b044bbf04f4c-kube-api-access-xxscw\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jscm6\" (UID: \"21b86bc7-8a12-4fdc-9074-b044bbf04f4c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jscm6"
Oct 30 20:32:49.855200 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:49.855140    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21b86bc7-8a12-4fdc-9074-b044bbf04f4c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jscm6\" (UID: \"21b86bc7-8a12-4fdc-9074-b044bbf04f4c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jscm6"
Oct 30 20:32:49.855255 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:49.855240    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/21b86bc7-8a12-4fdc-9074-b044bbf04f4c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jscm6\" (UID: \"21b86bc7-8a12-4fdc-9074-b044bbf04f4c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jscm6"
Oct 30 20:32:49.855286 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:49.855266    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/21b86bc7-8a12-4fdc-9074-b044bbf04f4c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jscm6\" (UID: \"21b86bc7-8a12-4fdc-9074-b044bbf04f4c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jscm6"
Oct 30 20:32:49.855321 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:49.855299    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/21b86bc7-8a12-4fdc-9074-b044bbf04f4c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jscm6\" (UID: \"21b86bc7-8a12-4fdc-9074-b044bbf04f4c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jscm6"
Oct 30 20:32:49.893221 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:49.893190    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jmh9r\""
Oct 30 20:32:49.895082 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:49.895062    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jscm6"]
Oct 30 20:32:49.955945 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:49.955905    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21b86bc7-8a12-4fdc-9074-b044bbf04f4c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jscm6\" (UID: \"21b86bc7-8a12-4fdc-9074-b044bbf04f4c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jscm6"
Oct 30 20:32:49.956111 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:49.955966    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/21b86bc7-8a12-4fdc-9074-b044bbf04f4c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jscm6\" (UID: \"21b86bc7-8a12-4fdc-9074-b044bbf04f4c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jscm6"
Oct 30 20:32:49.956111 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:49.955988    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/21b86bc7-8a12-4fdc-9074-b044bbf04f4c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jscm6\" (UID: \"21b86bc7-8a12-4fdc-9074-b044bbf04f4c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jscm6"
Oct 30 20:32:49.956111 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:49.956016    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/21b86bc7-8a12-4fdc-9074-b044bbf04f4c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jscm6\" (UID: \"21b86bc7-8a12-4fdc-9074-b044bbf04f4c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jscm6"
Oct 30 20:32:49.956238 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:49.956148    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/21b86bc7-8a12-4fdc-9074-b044bbf04f4c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jscm6\" (UID: \"21b86bc7-8a12-4fdc-9074-b044bbf04f4c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jscm6"
Oct 30 20:32:49.956288 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:49.956237    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/21b86bc7-8a12-4fdc-9074-b044bbf04f4c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jscm6\" (UID: \"21b86bc7-8a12-4fdc-9074-b044bbf04f4c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jscm6"
Oct 30 20:32:49.956288 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:49.956263    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxscw\" (UniqueName: \"kubernetes.io/projected/21b86bc7-8a12-4fdc-9074-b044bbf04f4c-kube-api-access-xxscw\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jscm6\" (UID: \"21b86bc7-8a12-4fdc-9074-b044bbf04f4c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jscm6"
Oct 30 20:32:49.956749 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:49.956723    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/21b86bc7-8a12-4fdc-9074-b044bbf04f4c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jscm6\" (UID: \"21b86bc7-8a12-4fdc-9074-b044bbf04f4c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jscm6"
Oct 30 20:32:49.958746 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:49.958716    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/21b86bc7-8a12-4fdc-9074-b044bbf04f4c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jscm6\" (UID: \"21b86bc7-8a12-4fdc-9074-b044bbf04f4c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jscm6"
Oct 30 20:32:49.958936 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:49.958904    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/21b86bc7-8a12-4fdc-9074-b044bbf04f4c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jscm6\" (UID: \"21b86bc7-8a12-4fdc-9074-b044bbf04f4c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jscm6"
Oct 30 20:32:49.959022 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:49.958988    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/21b86bc7-8a12-4fdc-9074-b044bbf04f4c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jscm6\" (UID: \"21b86bc7-8a12-4fdc-9074-b044bbf04f4c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jscm6"
Oct 30 20:32:49.959061 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:49.959028    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/21b86bc7-8a12-4fdc-9074-b044bbf04f4c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jscm6\" (UID: \"21b86bc7-8a12-4fdc-9074-b044bbf04f4c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jscm6"
Oct 30 20:32:50.015245 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:50.015218    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21b86bc7-8a12-4fdc-9074-b044bbf04f4c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jscm6\" (UID: \"21b86bc7-8a12-4fdc-9074-b044bbf04f4c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jscm6"
Oct 30 20:32:50.015396 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:50.015252    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxscw\" (UniqueName: \"kubernetes.io/projected/21b86bc7-8a12-4fdc-9074-b044bbf04f4c-kube-api-access-xxscw\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jscm6\" (UID: \"21b86bc7-8a12-4fdc-9074-b044bbf04f4c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jscm6"
Oct 30 20:32:50.098443 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:50.098415    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jscm6"
Oct 30 20:32:50.522805 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:50.522773    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jscm6"]
Oct 30 20:32:50.523798 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:32:50.523775    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21b86bc7_8a12_4fdc_9074_b044bbf04f4c.slice/crio-791eb5d53394ae53ca69cf568803f8841dfd518d95b3a4be8430eb4bfe9c6a8e WatchSource:0}: Error finding container 791eb5d53394ae53ca69cf568803f8841dfd518d95b3a4be8430eb4bfe9c6a8e: Status 404 returned error can't find the container with id 791eb5d53394ae53ca69cf568803f8841dfd518d95b3a4be8430eb4bfe9c6a8e
Oct 30 20:32:51.103136 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:51.103093    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jscm6" event={"ID":"21b86bc7-8a12-4fdc-9074-b044bbf04f4c","Type":"ContainerStarted","Data":"791eb5d53394ae53ca69cf568803f8841dfd518d95b3a4be8430eb4bfe9c6a8e"}
Oct 30 20:32:52.058567 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:52.058536    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-9pj25"
Oct 30 20:32:54.069251 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:54.069218    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6565584cfb-w8rp4"
Oct 30 20:32:54.238290 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:54.238255    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-9pj25"]
Oct 30 20:32:54.238558 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:54.238500    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-9pj25" podUID="3730060c-0d34-4d0f-af06-dee152463807" containerName="manager" containerID="cri-o://255e160fa7112afc5cec753793f383db4e6766be8e4d88a102ffe74046295f9a" gracePeriod=30
Oct 30 20:32:54.473098 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:54.473066    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:32:54.473181 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:54.473129    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:32:54.702416 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:54.702392    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-9pj25"
Oct 30 20:32:54.801982 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:54.801948    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3730060c-0d34-4d0f-af06-dee152463807-webhook-cert\" (UniqueName: \"kubernetes.io/secret/3730060c-0d34-4d0f-af06-dee152463807-webhook-cert\") pod \"3730060c-0d34-4d0f-af06-dee152463807\" (UID: \"3730060c-0d34-4d0f-af06-dee152463807\") "
Oct 30 20:32:54.802179 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:54.802051    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3730060c-0d34-4d0f-af06-dee152463807-kube-api-access-dhn5t\" (UniqueName: \"kubernetes.io/projected/3730060c-0d34-4d0f-af06-dee152463807-kube-api-access-dhn5t\") pod \"3730060c-0d34-4d0f-af06-dee152463807\" (UID: \"3730060c-0d34-4d0f-af06-dee152463807\") "
Oct 30 20:32:54.802179 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:54.802118    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3730060c-0d34-4d0f-af06-dee152463807-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3730060c-0d34-4d0f-af06-dee152463807-metrics-certs\") pod \"3730060c-0d34-4d0f-af06-dee152463807\" (UID: \"3730060c-0d34-4d0f-af06-dee152463807\") "
Oct 30 20:32:54.804431 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:54.804404    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3730060c-0d34-4d0f-af06-dee152463807-kube-api-access-dhn5t" pod "3730060c-0d34-4d0f-af06-dee152463807" (UID: "3730060c-0d34-4d0f-af06-dee152463807"). InnerVolumeSpecName "kube-api-access-dhn5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:32:54.804569 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:54.804512    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3730060c-0d34-4d0f-af06-dee152463807-webhook-cert" pod "3730060c-0d34-4d0f-af06-dee152463807" (UID: "3730060c-0d34-4d0f-af06-dee152463807"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:32:54.804702 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:54.804677    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3730060c-0d34-4d0f-af06-dee152463807-metrics-certs" pod "3730060c-0d34-4d0f-af06-dee152463807" (UID: "3730060c-0d34-4d0f-af06-dee152463807"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:32:54.873184 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:54.873146    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:54.903557 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:54.903526    5071 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3730060c-0d34-4d0f-af06-dee152463807-metrics-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:32:54.903557 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:54.903555    5071 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3730060c-0d34-4d0f-af06-dee152463807-webhook-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:32:54.903736 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:54.903568    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhn5t\" (UniqueName: \"kubernetes.io/projected/3730060c-0d34-4d0f-af06-dee152463807-kube-api-access-dhn5t\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:32:55.086255 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:55.086167    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:55.125369 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:55.125336    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jscm6" event={"ID":"21b86bc7-8a12-4fdc-9074-b044bbf04f4c","Type":"ContainerStarted","Data":"43575e9c605c437a69a9c6b63ccb07ef499cbf6fdec97b8559dcbba48a5efa44"}
Oct 30 20:32:55.125526 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:55.125404    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jscm6"
Oct 30 20:32:55.127577 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:55.127547    5071 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-jscm6 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:32:55.127750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:55.127605    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jscm6" podUID="21b86bc7-8a12-4fdc-9074-b044bbf04f4c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:32:55.128969 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:55.128935    5071 generic.go:356] "Generic (PLEG): container finished" podID="3730060c-0d34-4d0f-af06-dee152463807" containerID="255e160fa7112afc5cec753793f383db4e6766be8e4d88a102ffe74046295f9a" exitCode=0
Oct 30 20:32:55.129098 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:55.129062    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-9pj25" event={"ID":"3730060c-0d34-4d0f-af06-dee152463807","Type":"ContainerDied","Data":"255e160fa7112afc5cec753793f383db4e6766be8e4d88a102ffe74046295f9a"}
Oct 30 20:32:55.129161 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:55.129112    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-9pj25" event={"ID":"3730060c-0d34-4d0f-af06-dee152463807","Type":"ContainerDied","Data":"7da6ba73780e9718f6e3a7c032267b227fa4e30cae4970523ce6eba2dd05a0bf"}
Oct 30 20:32:55.129161 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:55.129144    5071 scope.go:122] "RemoveContainer" containerID="255e160fa7112afc5cec753793f383db4e6766be8e4d88a102ffe74046295f9a"
Oct 30 20:32:55.129372 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:55.129350    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-9pj25"
Oct 30 20:32:55.146925 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:55.146901    5071 scope.go:122] "RemoveContainer" containerID="255e160fa7112afc5cec753793f383db4e6766be8e4d88a102ffe74046295f9a"
Oct 30 20:32:55.147305 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:32:55.147280    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"255e160fa7112afc5cec753793f383db4e6766be8e4d88a102ffe74046295f9a\": container with ID starting with 255e160fa7112afc5cec753793f383db4e6766be8e4d88a102ffe74046295f9a not found: ID does not exist" containerID="255e160fa7112afc5cec753793f383db4e6766be8e4d88a102ffe74046295f9a"
Oct 30 20:32:55.147384 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:55.147319    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"255e160fa7112afc5cec753793f383db4e6766be8e4d88a102ffe74046295f9a"} err="failed to get container status \"255e160fa7112afc5cec753793f383db4e6766be8e4d88a102ffe74046295f9a\": rpc error: code = NotFound desc = could not find container \"255e160fa7112afc5cec753793f383db4e6766be8e4d88a102ffe74046295f9a\": container with ID starting with 255e160fa7112afc5cec753793f383db4e6766be8e4d88a102ffe74046295f9a not found: ID does not exist"
Oct 30 20:32:55.246454 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:55.246400    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jscm6" podStartSLOduration=2.2999591600000002 podStartE2EDuration="6.246386169s" totalImagesPullingTime="3.946427009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:32:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:32:55.24480724 +0000 UTC m=+387.161828770" watchObservedRunningTime="2026-10-30 20:32:55.246386169 +0000 UTC m=+387.163407696"
Oct 30 20:32:55.254166 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:55.254137    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:55.358851 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:55.358749    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-9pj25"]
Oct 30 20:32:55.365179 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:55.364948    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-9pj25"]
Oct 30 20:32:55.427553 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:55.427519    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:55.584536 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:55.584502    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:55.752961 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:55.752870    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:55.915798 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:55.915765    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:56.086364 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:56.086331    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:32:56.136020 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:56.135996    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jscm6"
Oct 30 20:32:56.199783 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:56.199753    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3730060c-0d34-4d0f-af06-dee152463807" path="/var/lib/kubelet/pods/3730060c-0d34-4d0f-af06-dee152463807/volumes"
Oct 30 20:32:57.156043 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:32:57.155959    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:02.360021 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:02.359975    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:02.514573 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:02.514543    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:02.669078 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:02.668996    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:02.833387 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:02.833349    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:02.991004 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:02.990909    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:03.151117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:03.151081    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:03.341411 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:03.341379    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:03.502133 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:03.502098    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:03.663358 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:03.663275    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:11.834435 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:11.834344    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:12.013606 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:12.013572    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:13.941247 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:13.941208    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:14.170140 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:14.170070    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:14.337523 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:14.337489    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:16.728216 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:33:16.728177    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:33:20.465417 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:20.465379    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:20.622566 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:20.622538    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:33:21.608538 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:21.608499    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:23.621632 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:23.621591    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:26.147015 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:26.146979    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:26.426253 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:26.426166    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:33:30.310525 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:30.310492    5071 generic.go:356] "Generic (PLEG): container finished" podID="1326fe90-f3ab-4922-8ab9-a6eab1dcf665" containerID="6f4b4ac41dc3c538c0bc81f7f4833c35ab5de48c80a76db4e1ed45a70038a4a1" exitCode=0
Oct 30 20:33:30.311031 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:30.310576    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7" event={"ID":"1326fe90-f3ab-4922-8ab9-a6eab1dcf665","Type":"ContainerDied","Data":"6f4b4ac41dc3c538c0bc81f7f4833c35ab5de48c80a76db4e1ed45a70038a4a1"}
Oct 30 20:33:30.311031 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:30.310962    5071 scope.go:122] "RemoveContainer" containerID="6f4b4ac41dc3c538c0bc81f7f4833c35ab5de48c80a76db4e1ed45a70038a4a1"
Oct 30 20:33:30.312165 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:30.312129    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:33:30.312261 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:30.312205    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:33:30.312261 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:30.312237    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:33:31.317731 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:31.317702    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7" event={"ID":"1326fe90-f3ab-4922-8ab9-a6eab1dcf665","Type":"ContainerStarted","Data":"c127aec28f417408eb2ad964c94910dec03e8f8ae6778b2720b79dd0a3951de9"}
Oct 30 20:33:31.341844 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:31.337186    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:33:31.343538 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:31.343192    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-9lhz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Oct 30 20:33:31.343538 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:31.343253    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7" podUID="1326fe90-f3ab-4922-8ab9-a6eab1dcf665" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Oct 30 20:33:32.349849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:32.343210    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-9lhz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Oct 30 20:33:32.349849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:32.343272    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7" podUID="1326fe90-f3ab-4922-8ab9-a6eab1dcf665" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Oct 30 20:33:33.347848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:33.345224    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-9lhz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Oct 30 20:33:33.347848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:33.345301    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7" podUID="1326fe90-f3ab-4922-8ab9-a6eab1dcf665" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Oct 30 20:33:34.349849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:34.342920    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-9lhz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Oct 30 20:33:34.349849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:34.342989    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7" podUID="1326fe90-f3ab-4922-8ab9-a6eab1dcf665" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Oct 30 20:33:35.326494 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:35.326460    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:33:35.349569 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:35.349534    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-9lhz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Oct 30 20:33:35.349879 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:35.349847    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7" podUID="1326fe90-f3ab-4922-8ab9-a6eab1dcf665" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Oct 30 20:33:36.342240 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:36.342205    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-9lhz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Oct 30 20:33:36.342536 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:36.342505    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7" podUID="1326fe90-f3ab-4922-8ab9-a6eab1dcf665" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Oct 30 20:33:37.340121 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:37.340089    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:33:37.340957 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:37.340938    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:33:46.728082 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:33:46.728046    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:33:49.886612 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:33:49.886565    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:34:16.727228 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:34:16.727187    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:34:20.804118 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:34:20.804071    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:34:21.941315 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:34:21.941283    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:34:23.000364 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:34:23.000334    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:34:23.182358 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:34:23.182315    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:34:23.941801 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:34:23.941759    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:34:40.172108 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:34:40.171908    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:34:40.343545 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:34:40.343509    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:34:45.270659 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:34:45.270622    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:34:45.589804 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:34:45.589769    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:34:46.726963 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:34:46.726924    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:35:06.902226 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:06.902189    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:35:09.807234 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:09.807196    5071 generic.go:356] "Generic (PLEG): container finished" podID="b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607" containerID="d1edfe6aec2489767e8aa2e1c677191ae5cc7c382ab8e800953737d207cf3e61" exitCode=1
Oct 30 20:35:09.807653 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:09.807265    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6565584cfb-w8rp4" event={"ID":"b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607","Type":"ContainerDied","Data":"d1edfe6aec2489767e8aa2e1c677191ae5cc7c382ab8e800953737d207cf3e61"}
Oct 30 20:35:09.807653 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:09.807606    5071 scope.go:122] "RemoveContainer" containerID="d1edfe6aec2489767e8aa2e1c677191ae5cc7c382ab8e800953737d207cf3e61"
Oct 30 20:35:10.388363 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:10.388310    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:35:10.814307 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:10.814274    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6565584cfb-w8rp4" event={"ID":"b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607","Type":"ContainerStarted","Data":"e88f539c1d7621283f4029f206bdb6d628705eba5a59a3b5d965f29a1368ba4a"}
Oct 30 20:35:10.814718 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:10.814495    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6565584cfb-w8rp4"
Oct 30 20:35:13.167471 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:13.167436    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:35:16.727545 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:35:16.727510    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:35:17.607861 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:17.607805    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:35:20.278645 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:20.278608    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:35:20.439639 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:20.439597    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:35:21.141195 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:21.141159    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:35:21.508282 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:21.508203    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:35:21.821640 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:21.821608    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6565584cfb-w8rp4"
Oct 30 20:35:22.277985 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:22.277871    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:35:22.496249 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:22.496187    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:35:24.289977 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:24.289945    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:35:26.387895 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:26.387857    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:35:29.534151 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:29.534119    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:35:32.055122 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:32.055091    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:35:33.669248 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:33.669204    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds"]
Oct 30 20:35:33.669862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:33.669842    5071 cpu_manager.go:414] "containerMap: removing container" podUID="3730060c-0d34-4d0f-af06-dee152463807" containerName="manager"
Oct 30 20:35:33.669961 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:33.669864    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3730060c-0d34-4d0f-af06-dee152463807" containerName="manager"
Oct 30 20:35:33.670023 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:33.669989    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="3730060c-0d34-4d0f-af06-dee152463807" containerName="manager"
Oct 30 20:35:33.673789 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:33.673755    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds"
Oct 30 20:35:33.732359 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:33.732333    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds"]
Oct 30 20:35:33.773623 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:33.773592    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:35:33.779312 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:33.779286    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-zzf7l\""
Oct 30 20:35:33.779312 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:33.779288    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 20:35:33.779481 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:33.779292    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 20:35:33.779481 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:33.779362    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 30 20:35:33.781664 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:33.781645    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b24c7ce-f038-48b4-bc61-e1b6040b8e83-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds\" (UID: \"5b24c7ce-f038-48b4-bc61-e1b6040b8e83\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds"
Oct 30 20:35:33.781751 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:33.781686    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpns8\" (UniqueName: \"kubernetes.io/projected/5b24c7ce-f038-48b4-bc61-e1b6040b8e83-kube-api-access-xpns8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds\" (UID: \"5b24c7ce-f038-48b4-bc61-e1b6040b8e83\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds"
Oct 30 20:35:33.781751 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:33.781718    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b24c7ce-f038-48b4-bc61-e1b6040b8e83-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds\" (UID: \"5b24c7ce-f038-48b4-bc61-e1b6040b8e83\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds"
Oct 30 20:35:33.781868 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:33.781854    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b24c7ce-f038-48b4-bc61-e1b6040b8e83-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds\" (UID: \"5b24c7ce-f038-48b4-bc61-e1b6040b8e83\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds"
Oct 30 20:35:33.781919 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:33.781885    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b24c7ce-f038-48b4-bc61-e1b6040b8e83-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds\" (UID: \"5b24c7ce-f038-48b4-bc61-e1b6040b8e83\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds"
Oct 30 20:35:33.782006 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:33.781994    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b24c7ce-f038-48b4-bc61-e1b6040b8e83-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds\" (UID: \"5b24c7ce-f038-48b4-bc61-e1b6040b8e83\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds"
Oct 30 20:35:33.882918 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:33.882887    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpns8\" (UniqueName: \"kubernetes.io/projected/5b24c7ce-f038-48b4-bc61-e1b6040b8e83-kube-api-access-xpns8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds\" (UID: \"5b24c7ce-f038-48b4-bc61-e1b6040b8e83\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds"
Oct 30 20:35:33.883123 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:33.882938    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b24c7ce-f038-48b4-bc61-e1b6040b8e83-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds\" (UID: \"5b24c7ce-f038-48b4-bc61-e1b6040b8e83\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds"
Oct 30 20:35:33.883123 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:33.882961    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b24c7ce-f038-48b4-bc61-e1b6040b8e83-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds\" (UID: \"5b24c7ce-f038-48b4-bc61-e1b6040b8e83\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds"
Oct 30 20:35:33.883123 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:33.882993    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b24c7ce-f038-48b4-bc61-e1b6040b8e83-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds\" (UID: \"5b24c7ce-f038-48b4-bc61-e1b6040b8e83\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds"
Oct 30 20:35:33.883123 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:33.883051    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b24c7ce-f038-48b4-bc61-e1b6040b8e83-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds\" (UID: \"5b24c7ce-f038-48b4-bc61-e1b6040b8e83\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds"
Oct 30 20:35:33.883123 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:33.883083    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b24c7ce-f038-48b4-bc61-e1b6040b8e83-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds\" (UID: \"5b24c7ce-f038-48b4-bc61-e1b6040b8e83\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds"
Oct 30 20:35:33.883489 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:33.883463    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b24c7ce-f038-48b4-bc61-e1b6040b8e83-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds\" (UID: \"5b24c7ce-f038-48b4-bc61-e1b6040b8e83\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds"
Oct 30 20:35:33.883625 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:33.883495    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b24c7ce-f038-48b4-bc61-e1b6040b8e83-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds\" (UID: \"5b24c7ce-f038-48b4-bc61-e1b6040b8e83\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds"
Oct 30 20:35:33.883625 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:33.883520    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b24c7ce-f038-48b4-bc61-e1b6040b8e83-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds\" (UID: \"5b24c7ce-f038-48b4-bc61-e1b6040b8e83\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds"
Oct 30 20:35:33.885432 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:33.885412    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b24c7ce-f038-48b4-bc61-e1b6040b8e83-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds\" (UID: \"5b24c7ce-f038-48b4-bc61-e1b6040b8e83\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds"
Oct 30 20:35:33.885582 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:33.885565    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b24c7ce-f038-48b4-bc61-e1b6040b8e83-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds\" (UID: \"5b24c7ce-f038-48b4-bc61-e1b6040b8e83\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds"
Oct 30 20:35:33.941186 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:33.941119    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpns8\" (UniqueName: \"kubernetes.io/projected/5b24c7ce-f038-48b4-bc61-e1b6040b8e83-kube-api-access-xpns8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds\" (UID: \"5b24c7ce-f038-48b4-bc61-e1b6040b8e83\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds"
Oct 30 20:35:33.990873 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:33.990839    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds"
Oct 30 20:35:34.374478 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:34.374446    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds"]
Oct 30 20:35:34.376249 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:35:34.376218    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b24c7ce_f038_48b4_bc61_e1b6040b8e83.slice/crio-4baf6da1b427941f868e6e5ee596a95d2af39edde4667ebe0191115840cc9693 WatchSource:0}: Error finding container 4baf6da1b427941f868e6e5ee596a95d2af39edde4667ebe0191115840cc9693: Status 404 returned error can't find the container with id 4baf6da1b427941f868e6e5ee596a95d2af39edde4667ebe0191115840cc9693
Oct 30 20:35:34.933552 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:34.933520    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds" event={"ID":"5b24c7ce-f038-48b4-bc61-e1b6040b8e83","Type":"ContainerStarted","Data":"4baf6da1b427941f868e6e5ee596a95d2af39edde4667ebe0191115840cc9693"}
Oct 30 20:35:36.934761 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:36.934719    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:35:36.945077 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:36.945043    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds" event={"ID":"5b24c7ce-f038-48b4-bc61-e1b6040b8e83","Type":"ContainerStarted","Data":"7cedbc3d7406458f0db3e306237380b598fb66737db455092de978c44cc5d27b"}
Oct 30 20:35:36.991577 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:36.991547    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds"
Oct 30 20:35:37.066144 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:37.066073    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds" podStartSLOduration=2.510317628 podStartE2EDuration="4.066053747s" totalImagesPullingTime="1.555736119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:35:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:35:37.063896625 +0000 UTC m=+548.980918161" watchObservedRunningTime="2026-10-30 20:35:37.066053747 +0000 UTC m=+548.983075284"
Oct 30 20:35:43.273363 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:43.273320    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:35:43.991313 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:43.991271    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds"
Oct 30 20:35:44.003979 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:44.003950    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds"
Oct 30 20:35:44.994510 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:44.994482    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds"
Oct 30 20:35:45.790560 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:45.790528    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth"]
Oct 30 20:35:45.794144 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:45.794124    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth"
Oct 30 20:35:45.888776 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:45.888741    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth"]
Oct 30 20:35:45.893360 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:45.893334    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecd40162-84fb-4838-ae3b-bafa49f9cdb9-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6hth\" (UID: \"ecd40162-84fb-4838-ae3b-bafa49f9cdb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth"
Oct 30 20:35:45.893519 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:45.893394    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecd40162-84fb-4838-ae3b-bafa49f9cdb9-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6hth\" (UID: \"ecd40162-84fb-4838-ae3b-bafa49f9cdb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth"
Oct 30 20:35:45.893519 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:45.893486    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecd40162-84fb-4838-ae3b-bafa49f9cdb9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6hth\" (UID: \"ecd40162-84fb-4838-ae3b-bafa49f9cdb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth"
Oct 30 20:35:45.893647 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:45.893527    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24nw6\" (UniqueName: \"kubernetes.io/projected/ecd40162-84fb-4838-ae3b-bafa49f9cdb9-kube-api-access-24nw6\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6hth\" (UID: \"ecd40162-84fb-4838-ae3b-bafa49f9cdb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth"
Oct 30 20:35:45.893647 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:45.893607    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd40162-84fb-4838-ae3b-bafa49f9cdb9-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6hth\" (UID: \"ecd40162-84fb-4838-ae3b-bafa49f9cdb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth"
Oct 30 20:35:45.893647 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:45.893643    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd40162-84fb-4838-ae3b-bafa49f9cdb9-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6hth\" (UID: \"ecd40162-84fb-4838-ae3b-bafa49f9cdb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth"
Oct 30 20:35:45.893802 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:45.893673    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecd40162-84fb-4838-ae3b-bafa49f9cdb9-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6hth\" (UID: \"ecd40162-84fb-4838-ae3b-bafa49f9cdb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth"
Oct 30 20:35:45.893802 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:45.893747    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecd40162-84fb-4838-ae3b-bafa49f9cdb9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6hth\" (UID: \"ecd40162-84fb-4838-ae3b-bafa49f9cdb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth"
Oct 30 20:35:45.893802 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:45.893777    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecd40162-84fb-4838-ae3b-bafa49f9cdb9-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6hth\" (UID: \"ecd40162-84fb-4838-ae3b-bafa49f9cdb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth"
Oct 30 20:35:45.995275 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:45.995245    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecd40162-84fb-4838-ae3b-bafa49f9cdb9-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6hth\" (UID: \"ecd40162-84fb-4838-ae3b-bafa49f9cdb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth"
Oct 30 20:35:45.995275 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:45.995277    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecd40162-84fb-4838-ae3b-bafa49f9cdb9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6hth\" (UID: \"ecd40162-84fb-4838-ae3b-bafa49f9cdb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth"
Oct 30 20:35:45.995808 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:45.995293    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24nw6\" (UniqueName: \"kubernetes.io/projected/ecd40162-84fb-4838-ae3b-bafa49f9cdb9-kube-api-access-24nw6\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6hth\" (UID: \"ecd40162-84fb-4838-ae3b-bafa49f9cdb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth"
Oct 30 20:35:45.995808 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:45.995315    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd40162-84fb-4838-ae3b-bafa49f9cdb9-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6hth\" (UID: \"ecd40162-84fb-4838-ae3b-bafa49f9cdb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth"
Oct 30 20:35:45.995808 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:45.995340    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd40162-84fb-4838-ae3b-bafa49f9cdb9-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6hth\" (UID: \"ecd40162-84fb-4838-ae3b-bafa49f9cdb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth"
Oct 30 20:35:45.995808 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:45.995378    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecd40162-84fb-4838-ae3b-bafa49f9cdb9-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6hth\" (UID: \"ecd40162-84fb-4838-ae3b-bafa49f9cdb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth"
Oct 30 20:35:45.995808 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:45.995410    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecd40162-84fb-4838-ae3b-bafa49f9cdb9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6hth\" (UID: \"ecd40162-84fb-4838-ae3b-bafa49f9cdb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth"
Oct 30 20:35:45.995808 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:45.995436    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecd40162-84fb-4838-ae3b-bafa49f9cdb9-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6hth\" (UID: \"ecd40162-84fb-4838-ae3b-bafa49f9cdb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth"
Oct 30 20:35:45.995808 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:45.995514    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecd40162-84fb-4838-ae3b-bafa49f9cdb9-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6hth\" (UID: \"ecd40162-84fb-4838-ae3b-bafa49f9cdb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth"
Oct 30 20:35:45.996198 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:45.995967    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd40162-84fb-4838-ae3b-bafa49f9cdb9-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6hth\" (UID: \"ecd40162-84fb-4838-ae3b-bafa49f9cdb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth"
Oct 30 20:35:45.996198 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:45.996053    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecd40162-84fb-4838-ae3b-bafa49f9cdb9-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6hth\" (UID: \"ecd40162-84fb-4838-ae3b-bafa49f9cdb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth"
Oct 30 20:35:45.996198 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:45.996141    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecd40162-84fb-4838-ae3b-bafa49f9cdb9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6hth\" (UID: \"ecd40162-84fb-4838-ae3b-bafa49f9cdb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth"
Oct 30 20:35:45.996316 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:45.996221    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecd40162-84fb-4838-ae3b-bafa49f9cdb9-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6hth\" (UID: \"ecd40162-84fb-4838-ae3b-bafa49f9cdb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth"
Oct 30 20:35:45.996398 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:45.996381    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecd40162-84fb-4838-ae3b-bafa49f9cdb9-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6hth\" (UID: \"ecd40162-84fb-4838-ae3b-bafa49f9cdb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth"
Oct 30 20:35:45.998418 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:45.998391    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecd40162-84fb-4838-ae3b-bafa49f9cdb9-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6hth\" (UID: \"ecd40162-84fb-4838-ae3b-bafa49f9cdb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth"
Oct 30 20:35:45.998610 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:45.998593    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecd40162-84fb-4838-ae3b-bafa49f9cdb9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6hth\" (UID: \"ecd40162-84fb-4838-ae3b-bafa49f9cdb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth"
Oct 30 20:35:46.058083 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:46.058017    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecd40162-84fb-4838-ae3b-bafa49f9cdb9-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6hth\" (UID: \"ecd40162-84fb-4838-ae3b-bafa49f9cdb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth"
Oct 30 20:35:46.058362 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:46.058337    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24nw6\" (UniqueName: \"kubernetes.io/projected/ecd40162-84fb-4838-ae3b-bafa49f9cdb9-kube-api-access-24nw6\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s6hth\" (UID: \"ecd40162-84fb-4838-ae3b-bafa49f9cdb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth"
Oct 30 20:35:46.106339 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:46.106314    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth"
Oct 30 20:35:46.529149 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:46.529123    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth"]
Oct 30 20:35:46.531339 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:35:46.531310    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd40162_84fb_4838_ae3b_bafa49f9cdb9.slice/crio-ad0d8870ee09d8a8a6ff55c90e40e81d6b741f4882daa26423810d016add41d2 WatchSource:0}: Error finding container ad0d8870ee09d8a8a6ff55c90e40e81d6b741f4882daa26423810d016add41d2: Status 404 returned error can't find the container with id ad0d8870ee09d8a8a6ff55c90e40e81d6b741f4882daa26423810d016add41d2
Oct 30 20:35:46.534147 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:46.534106    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:35:46.534224 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:46.534200    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:35:46.534264 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:46.534243    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:35:46.727011 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:35:46.726977    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:35:46.997409 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:46.997377    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth" event={"ID":"ecd40162-84fb-4838-ae3b-bafa49f9cdb9","Type":"ContainerStarted","Data":"16ef5b8853fdb09b090148a2f559c203878667fd4be7ffd6a290f239d24b2451"}
Oct 30 20:35:46.997943 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:46.997439    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth" event={"ID":"ecd40162-84fb-4838-ae3b-bafa49f9cdb9","Type":"ContainerStarted","Data":"ad0d8870ee09d8a8a6ff55c90e40e81d6b741f4882daa26423810d016add41d2"}
Oct 30 20:35:47.106616 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:47.106580    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth"
Oct 30 20:35:47.108274 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:47.108248    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-s6hth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 20:35:47.108403 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:47.108307    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth" podUID="ecd40162-84fb-4838-ae3b-bafa49f9cdb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 20:35:47.140861 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:47.139402    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth" podStartSLOduration=2.139381867 podStartE2EDuration="2.139381867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:35:47.132278457 +0000 UTC m=+559.049300006" watchObservedRunningTime="2026-10-30 20:35:47.139381867 +0000 UTC m=+559.056403404"
Oct 30 20:35:48.107666 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:48.107633    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-s6hth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 20:35:48.108188 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:48.107703    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth" podUID="ecd40162-84fb-4838-ae3b-bafa49f9cdb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 20:35:49.107020 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:49.106976    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-s6hth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 20:35:49.107234 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:49.107046    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth" podUID="ecd40162-84fb-4838-ae3b-bafa49f9cdb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 20:35:50.107659 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:50.107618    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-s6hth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 20:35:50.108089 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:50.107711    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth" podUID="ecd40162-84fb-4838-ae3b-bafa49f9cdb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 20:35:51.107051 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:51.107016    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-s6hth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 20:35:51.107231 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:51.107071    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth" podUID="ecd40162-84fb-4838-ae3b-bafa49f9cdb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 20:35:52.107061 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:52.107023    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-s6hth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 20:35:52.107485 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:52.107088    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth" podUID="ecd40162-84fb-4838-ae3b-bafa49f9cdb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 20:35:53.106986 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:53.106953    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-s6hth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 20:35:53.107153 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:53.107008    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth" podUID="ecd40162-84fb-4838-ae3b-bafa49f9cdb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 20:35:55.107314 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:55.107264    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-s6hth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:35:55.107837 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:55.107368    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth" podUID="ecd40162-84fb-4838-ae3b-bafa49f9cdb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:35:56.148979 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:56.148929    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-s6hth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:35:56.149502 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:56.148989    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth" podUID="ecd40162-84fb-4838-ae3b-bafa49f9cdb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:35:56.256290 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:56.256257    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth"
Oct 30 20:35:56.256624 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:56.256594    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth"
Oct 30 20:35:56.257378 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:56.257361    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth"
Oct 30 20:35:58.770310 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:35:58.770274    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:36:04.798572 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:36:04.798533    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:36:05.148905 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:36:05.148797    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:36:12.354271 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:36:12.354191    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:36:12.523264 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:36:12.523226    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:36:16.727973 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:36:16.727930    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:36:17.957251 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:36:17.957212    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:36:21.487081 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:36:21.487047    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:36:22.617181 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:36:22.617148    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:36:24.630980 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:36:24.630946    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:36:28.255325 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:36:28.255297    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8qvb_91fe9e06-0c30-4000-bddb-2ba89cd048cc/console-operator/2.log"
Oct 30 20:36:28.256618 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:36:28.256597    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8qvb_91fe9e06-0c30-4000-bddb-2ba89cd048cc/console-operator/2.log"
Oct 30 20:36:28.263461 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:36:28.263441    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q4f4h_80cae79e-b60f-4efe-bb58-e77753a457e5/ovn-acl-logging/0.log"
Oct 30 20:36:28.265061 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:36:28.265041    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q4f4h_80cae79e-b60f-4efe-bb58-e77753a457e5/ovn-acl-logging/0.log"
Oct 30 20:36:34.366154 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:36:34.366117    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:36:34.831967 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:36:34.831930    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:36:36.441213 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:36:36.441176    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:36:36.606288 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:36:36.606245    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:36:38.130833 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:36:38.130782    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:36:38.301195 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:36:38.301159    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:36:46.540471 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:36:46.540427    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:36:46.707738 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:36:46.707702    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:36:46.729033 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:36:46.728994    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:37:09.337759 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:09.337723    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:09.512299 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:09.512267    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:10.063195 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:10.063153    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:10.227812 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:10.227775    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:10.412216 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:10.412124    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:10.594806 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:10.594777    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:10.758583 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:10.758497    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:11.005946 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:11.005905    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:11.200506 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:11.200468    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:11.377310 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:11.377276    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:11.549783 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:11.549740    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:11.704294 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:11.704252    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"]
Oct 30 20:37:11.707880 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:11.707858    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:37:11.761609 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:11.761576    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-b43c84-openshift-default-dockercfg-xpbj6\""
Oct 30 20:37:11.811736 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:11.811653    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5863cbee-17a0-4b2e-a6e5-c46737b72491-istiod-ca-cert\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d\" (UID: \"5863cbee-17a0-4b2e-a6e5-c46737b72491\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:37:11.811736 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:11.811702    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5863cbee-17a0-4b2e-a6e5-c46737b72491-istio-token\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d\" (UID: \"5863cbee-17a0-4b2e-a6e5-c46737b72491\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:37:11.811992 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:11.811883    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5863cbee-17a0-4b2e-a6e5-c46737b72491-credential-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d\" (UID: \"5863cbee-17a0-4b2e-a6e5-c46737b72491\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:37:11.811992 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:11.811924    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5863cbee-17a0-4b2e-a6e5-c46737b72491-workload-certs\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d\" (UID: \"5863cbee-17a0-4b2e-a6e5-c46737b72491\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:37:11.812103 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:11.812023    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5863cbee-17a0-4b2e-a6e5-c46737b72491-istio-data\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d\" (UID: \"5863cbee-17a0-4b2e-a6e5-c46737b72491\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:37:11.812103 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:11.812074    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5863cbee-17a0-4b2e-a6e5-c46737b72491-istio-podinfo\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d\" (UID: \"5863cbee-17a0-4b2e-a6e5-c46737b72491\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:37:11.812183 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:11.812115    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5863cbee-17a0-4b2e-a6e5-c46737b72491-istio-envoy\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d\" (UID: \"5863cbee-17a0-4b2e-a6e5-c46737b72491\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:37:11.812216 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:11.812188    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zc2l\" (UniqueName: \"kubernetes.io/projected/5863cbee-17a0-4b2e-a6e5-c46737b72491-kube-api-access-2zc2l\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d\" (UID: \"5863cbee-17a0-4b2e-a6e5-c46737b72491\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:37:11.812250 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:11.812217    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5863cbee-17a0-4b2e-a6e5-c46737b72491-workload-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d\" (UID: \"5863cbee-17a0-4b2e-a6e5-c46737b72491\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:37:11.812314 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:11.812296    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"]
Oct 30 20:37:11.841079 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:11.841052    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:11.912896 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:11.912859    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5863cbee-17a0-4b2e-a6e5-c46737b72491-istiod-ca-cert\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d\" (UID: \"5863cbee-17a0-4b2e-a6e5-c46737b72491\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:37:11.913092 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:11.912914    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5863cbee-17a0-4b2e-a6e5-c46737b72491-istio-token\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d\" (UID: \"5863cbee-17a0-4b2e-a6e5-c46737b72491\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:37:11.913092 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:11.912970    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5863cbee-17a0-4b2e-a6e5-c46737b72491-credential-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d\" (UID: \"5863cbee-17a0-4b2e-a6e5-c46737b72491\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:37:11.913092 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:11.913001    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5863cbee-17a0-4b2e-a6e5-c46737b72491-workload-certs\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d\" (UID: \"5863cbee-17a0-4b2e-a6e5-c46737b72491\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:37:11.913092 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:11.913037    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5863cbee-17a0-4b2e-a6e5-c46737b72491-istio-data\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d\" (UID: \"5863cbee-17a0-4b2e-a6e5-c46737b72491\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:37:11.913092 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:11.913086    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5863cbee-17a0-4b2e-a6e5-c46737b72491-istio-podinfo\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d\" (UID: \"5863cbee-17a0-4b2e-a6e5-c46737b72491\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:37:11.913391 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:11.913121    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5863cbee-17a0-4b2e-a6e5-c46737b72491-istio-envoy\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d\" (UID: \"5863cbee-17a0-4b2e-a6e5-c46737b72491\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:37:11.913391 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:11.913158    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zc2l\" (UniqueName: \"kubernetes.io/projected/5863cbee-17a0-4b2e-a6e5-c46737b72491-kube-api-access-2zc2l\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d\" (UID: \"5863cbee-17a0-4b2e-a6e5-c46737b72491\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:37:11.913391 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:11.913190    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5863cbee-17a0-4b2e-a6e5-c46737b72491-workload-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d\" (UID: \"5863cbee-17a0-4b2e-a6e5-c46737b72491\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:37:11.913787 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:11.913699    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5863cbee-17a0-4b2e-a6e5-c46737b72491-istiod-ca-cert\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d\" (UID: \"5863cbee-17a0-4b2e-a6e5-c46737b72491\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:37:11.914022 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:11.913704    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5863cbee-17a0-4b2e-a6e5-c46737b72491-credential-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d\" (UID: \"5863cbee-17a0-4b2e-a6e5-c46737b72491\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:37:11.914022 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:11.913746    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5863cbee-17a0-4b2e-a6e5-c46737b72491-workload-certs\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d\" (UID: \"5863cbee-17a0-4b2e-a6e5-c46737b72491\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:37:11.914184 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:11.913774    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5863cbee-17a0-4b2e-a6e5-c46737b72491-istio-data\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d\" (UID: \"5863cbee-17a0-4b2e-a6e5-c46737b72491\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:37:11.914184 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:11.913812    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5863cbee-17a0-4b2e-a6e5-c46737b72491-workload-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d\" (UID: \"5863cbee-17a0-4b2e-a6e5-c46737b72491\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:37:11.915992 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:11.915961    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5863cbee-17a0-4b2e-a6e5-c46737b72491-istio-envoy\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d\" (UID: \"5863cbee-17a0-4b2e-a6e5-c46737b72491\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:37:11.916081 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:11.915974    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5863cbee-17a0-4b2e-a6e5-c46737b72491-istio-podinfo\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d\" (UID: \"5863cbee-17a0-4b2e-a6e5-c46737b72491\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:37:11.980500 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:11.980464    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5863cbee-17a0-4b2e-a6e5-c46737b72491-istio-token\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d\" (UID: \"5863cbee-17a0-4b2e-a6e5-c46737b72491\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:37:11.980627 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:11.980552    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zc2l\" (UniqueName: \"kubernetes.io/projected/5863cbee-17a0-4b2e-a6e5-c46737b72491-kube-api-access-2zc2l\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d\" (UID: \"5863cbee-17a0-4b2e-a6e5-c46737b72491\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:37:12.020195 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.020170    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:37:12.107769 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.107729    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"]
Oct 30 20:37:12.129756 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.129728    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:37:12.178640 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.178420    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"]
Oct 30 20:37:12.190205 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.189733    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-b32941-openshift-default-dockercfg-rlxkq\""
Oct 30 20:37:12.254205 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.253716    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:12.321807 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.321775    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99069007-e0f8-4092-8fd6-f27b0a119720-istio-data\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6\" (UID: \"99069007-e0f8-4092-8fd6-f27b0a119720\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:37:12.322019 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.321851    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99069007-e0f8-4092-8fd6-f27b0a119720-istio-podinfo\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6\" (UID: \"99069007-e0f8-4092-8fd6-f27b0a119720\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:37:12.322093 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.322023    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7frs\" (UniqueName: \"kubernetes.io/projected/99069007-e0f8-4092-8fd6-f27b0a119720-kube-api-access-h7frs\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6\" (UID: \"99069007-e0f8-4092-8fd6-f27b0a119720\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:37:12.322158 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.322113    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99069007-e0f8-4092-8fd6-f27b0a119720-istiod-ca-cert\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6\" (UID: \"99069007-e0f8-4092-8fd6-f27b0a119720\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:37:12.322217 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.322161    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99069007-e0f8-4092-8fd6-f27b0a119720-istio-envoy\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6\" (UID: \"99069007-e0f8-4092-8fd6-f27b0a119720\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:37:12.322354 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.322333    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99069007-e0f8-4092-8fd6-f27b0a119720-istio-token\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6\" (UID: \"99069007-e0f8-4092-8fd6-f27b0a119720\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:37:12.322420 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.322373    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99069007-e0f8-4092-8fd6-f27b0a119720-workload-certs\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6\" (UID: \"99069007-e0f8-4092-8fd6-f27b0a119720\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:37:12.322533 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.322452    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99069007-e0f8-4092-8fd6-f27b0a119720-workload-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6\" (UID: \"99069007-e0f8-4092-8fd6-f27b0a119720\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:37:12.322533 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.322499    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99069007-e0f8-4092-8fd6-f27b0a119720-credential-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6\" (UID: \"99069007-e0f8-4092-8fd6-f27b0a119720\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:37:12.423792 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.423718    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99069007-e0f8-4092-8fd6-f27b0a119720-istio-podinfo\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6\" (UID: \"99069007-e0f8-4092-8fd6-f27b0a119720\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:37:12.423792 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.423752    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7frs\" (UniqueName: \"kubernetes.io/projected/99069007-e0f8-4092-8fd6-f27b0a119720-kube-api-access-h7frs\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6\" (UID: \"99069007-e0f8-4092-8fd6-f27b0a119720\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:37:12.423792 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.423787    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99069007-e0f8-4092-8fd6-f27b0a119720-istiod-ca-cert\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6\" (UID: \"99069007-e0f8-4092-8fd6-f27b0a119720\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:37:12.424053 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.423878    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99069007-e0f8-4092-8fd6-f27b0a119720-istio-envoy\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6\" (UID: \"99069007-e0f8-4092-8fd6-f27b0a119720\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:37:12.424053 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.423935    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99069007-e0f8-4092-8fd6-f27b0a119720-istio-token\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6\" (UID: \"99069007-e0f8-4092-8fd6-f27b0a119720\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:37:12.424053 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.423961    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99069007-e0f8-4092-8fd6-f27b0a119720-workload-certs\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6\" (UID: \"99069007-e0f8-4092-8fd6-f27b0a119720\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:37:12.424053 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.423996    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99069007-e0f8-4092-8fd6-f27b0a119720-workload-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6\" (UID: \"99069007-e0f8-4092-8fd6-f27b0a119720\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:37:12.424053 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.424018    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99069007-e0f8-4092-8fd6-f27b0a119720-credential-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6\" (UID: \"99069007-e0f8-4092-8fd6-f27b0a119720\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:37:12.424300 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.424101    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99069007-e0f8-4092-8fd6-f27b0a119720-istio-data\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6\" (UID: \"99069007-e0f8-4092-8fd6-f27b0a119720\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:37:12.424354 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.424340    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99069007-e0f8-4092-8fd6-f27b0a119720-credential-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6\" (UID: \"99069007-e0f8-4092-8fd6-f27b0a119720\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:37:12.424531 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.424510    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99069007-e0f8-4092-8fd6-f27b0a119720-istio-data\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6\" (UID: \"99069007-e0f8-4092-8fd6-f27b0a119720\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:37:12.424604 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.424569    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99069007-e0f8-4092-8fd6-f27b0a119720-istiod-ca-cert\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6\" (UID: \"99069007-e0f8-4092-8fd6-f27b0a119720\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:37:12.424734 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.424684    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99069007-e0f8-4092-8fd6-f27b0a119720-workload-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6\" (UID: \"99069007-e0f8-4092-8fd6-f27b0a119720\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:37:12.425032 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.425007    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99069007-e0f8-4092-8fd6-f27b0a119720-workload-certs\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6\" (UID: \"99069007-e0f8-4092-8fd6-f27b0a119720\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:37:12.426419 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.426398    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99069007-e0f8-4092-8fd6-f27b0a119720-istio-envoy\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6\" (UID: \"99069007-e0f8-4092-8fd6-f27b0a119720\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:37:12.426511 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.426494    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99069007-e0f8-4092-8fd6-f27b0a119720-istio-podinfo\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6\" (UID: \"99069007-e0f8-4092-8fd6-f27b0a119720\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:37:12.432391 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.432367    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:12.453233 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.453216    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"]
Oct 30 20:37:12.454768 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:37:12.454748    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5863cbee_17a0_4b2e_a6e5_c46737b72491.slice/crio-af00e45e344752dbd2b3a62822999c705645ccfa620a539342d9acdceb744cb2 WatchSource:0}: Error finding container af00e45e344752dbd2b3a62822999c705645ccfa620a539342d9acdceb744cb2: Status 404 returned error can't find the container with id af00e45e344752dbd2b3a62822999c705645ccfa620a539342d9acdceb744cb2
Oct 30 20:37:12.456788 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.456759    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:37:12.456900 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.456883    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:37:12.456947 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.456922    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:37:12.482988 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.482965    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99069007-e0f8-4092-8fd6-f27b0a119720-istio-token\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6\" (UID: \"99069007-e0f8-4092-8fd6-f27b0a119720\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:37:12.483104 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.483041    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7frs\" (UniqueName: \"kubernetes.io/projected/99069007-e0f8-4092-8fd6-f27b0a119720-kube-api-access-h7frs\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6\" (UID: \"99069007-e0f8-4092-8fd6-f27b0a119720\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:37:12.599732 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.599677    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:12.747099 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:12.747016    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:37:13.174290 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:13.174251    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"]
Oct 30 20:37:13.175496 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:37:13.175470    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99069007_e0f8_4092_8fd6_f27b0a119720.slice/crio-3d9a041f137d200cfbbaddcd5be13196b399318015f7020ab1a331c3664520be WatchSource:0}: Error finding container 3d9a041f137d200cfbbaddcd5be13196b399318015f7020ab1a331c3664520be: Status 404 returned error can't find the container with id 3d9a041f137d200cfbbaddcd5be13196b399318015f7020ab1a331c3664520be
Oct 30 20:37:13.178013 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:13.177976    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:37:13.178098 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:13.178045    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:37:13.178098 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:13.178088    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:37:13.426570 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:13.426488    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6" event={"ID":"99069007-e0f8-4092-8fd6-f27b0a119720","Type":"ContainerStarted","Data":"63d8251f784c6fab20cf2a5ba32c42a286a091cc3a95f12eff1c80c5ddef01db"}
Oct 30 20:37:13.426570 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:13.426558    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6" event={"ID":"99069007-e0f8-4092-8fd6-f27b0a119720","Type":"ContainerStarted","Data":"3d9a041f137d200cfbbaddcd5be13196b399318015f7020ab1a331c3664520be"}
Oct 30 20:37:13.428865 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:13.428765    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d" event={"ID":"5863cbee-17a0-4b2e-a6e5-c46737b72491","Type":"ContainerStarted","Data":"fab3657d130ceced6ecc5f39b5a6707979abc9d0738657294ab14ba60df5fe37"}
Oct 30 20:37:13.428865 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:13.428850    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d" event={"ID":"5863cbee-17a0-4b2e-a6e5-c46737b72491","Type":"ContainerStarted","Data":"af00e45e344752dbd2b3a62822999c705645ccfa620a539342d9acdceb744cb2"}
Oct 30 20:37:13.550647 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:13.550590    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6" podStartSLOduration=1.550569181 podStartE2EDuration="1.550569181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:13.549657155 +0000 UTC m=+645.466678691" watchObservedRunningTime="2026-10-30 20:37:13.550569181 +0000 UTC m=+645.467590718"
Oct 30 20:37:13.700318 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:13.700215    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d" podStartSLOduration=2.700201232 podStartE2EDuration="2.700201232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:13.698467115 +0000 UTC m=+645.615488652" watchObservedRunningTime="2026-10-30 20:37:13.700201232 +0000 UTC m=+645.617222775"
Oct 30 20:37:13.747741 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:13.747713    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:37:14.020592 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:14.020501    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:37:14.025862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:14.025835    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:37:14.433569 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:14.433539    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:37:14.434464 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:14.434429    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:37:14.753111 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:14.753033    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:37:15.009117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:15.009031    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:15.174840 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:15.174780    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:15.346553 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:15.346508    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:15.440374 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:15.440338    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:37:15.441335 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:15.441302    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:37:15.661302 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:15.661212    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:15.925202 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:15.925123    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:16.106298 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:16.106262    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:16.490810 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:16.490781    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:16.672957 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:16.672927    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:16.732885 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:37:16.732849    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:37:16.930512 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:16.930471    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:17.347734 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:17.347704    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:17.589190 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:17.589110    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:17.762272 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:17.762180    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:17.993526 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:17.993490    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:18.180646 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:18.180607    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:18.341399 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:18.341361    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:20.257007 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:20.256971    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:20.525469 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:20.525317    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:20.656537 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:20.656508    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-86d620-fb8d7795f-mjmqg"]
Oct 30 20:37:20.661381 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:20.661365    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-86d620-fb8d7795f-mjmqg"
Oct 30 20:37:20.701830 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:20.701802    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdcth\" (UniqueName: \"kubernetes.io/projected/904e938e-ad13-4460-9ab6-b1ce6ed3c1d6-kube-api-access-jdcth\") pod \"payload-pre-processing-e2e-worker-w1-86d620-fb8d7795f-mjmqg\" (UID: \"904e938e-ad13-4460-9ab6-b1ce6ed3c1d6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-86d620-fb8d7795f-mjmqg"
Oct 30 20:37:20.701959 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:20.701869    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/904e938e-ad13-4460-9ab6-b1ce6ed3c1d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-86d620-fb8d7795f-mjmqg\" (UID: \"904e938e-ad13-4460-9ab6-b1ce6ed3c1d6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-86d620-fb8d7795f-mjmqg"
Oct 30 20:37:20.716421 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:20.716396    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-86d620\""
Oct 30 20:37:20.716555 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:20.716489    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-86d620-dockercfg-ntn49\""
Oct 30 20:37:20.770047 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:20.770005    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-86d620-fb8d7795f-mjmqg"]
Oct 30 20:37:20.800712 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:20.800661    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-86d620-74c5fd4b96-2zkm5"]
Oct 30 20:37:20.802891 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:20.802864    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdcth\" (UniqueName: \"kubernetes.io/projected/904e938e-ad13-4460-9ab6-b1ce6ed3c1d6-kube-api-access-jdcth\") pod \"payload-pre-processing-e2e-worker-w1-86d620-fb8d7795f-mjmqg\" (UID: \"904e938e-ad13-4460-9ab6-b1ce6ed3c1d6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-86d620-fb8d7795f-mjmqg"
Oct 30 20:37:20.803010 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:20.802910    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/904e938e-ad13-4460-9ab6-b1ce6ed3c1d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-86d620-fb8d7795f-mjmqg\" (UID: \"904e938e-ad13-4460-9ab6-b1ce6ed3c1d6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-86d620-fb8d7795f-mjmqg"
Oct 30 20:37:20.803867 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:20.803844    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/904e938e-ad13-4460-9ab6-b1ce6ed3c1d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-86d620-fb8d7795f-mjmqg\" (UID: \"904e938e-ad13-4460-9ab6-b1ce6ed3c1d6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-86d620-fb8d7795f-mjmqg"
Oct 30 20:37:20.804882 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:20.804862    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-86d620-74c5fd4b96-2zkm5"
Oct 30 20:37:20.869110 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:20.869077    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-86d620-74c5fd4b96-2zkm5"]
Oct 30 20:37:20.872233 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:20.872204    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdcth\" (UniqueName: \"kubernetes.io/projected/904e938e-ad13-4460-9ab6-b1ce6ed3c1d6-kube-api-access-jdcth\") pod \"payload-pre-processing-e2e-worker-w1-86d620-fb8d7795f-mjmqg\" (UID: \"904e938e-ad13-4460-9ab6-b1ce6ed3c1d6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-86d620-fb8d7795f-mjmqg"
Oct 30 20:37:20.903961 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:20.903923    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/514c415b-492e-4092-9889-c2835f58b761-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-86d620-74c5fd4b96-2zkm5\" (UID: \"514c415b-492e-4092-9889-c2835f58b761\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-86d620-74c5fd4b96-2zkm5"
Oct 30 20:37:20.904128 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:20.904111    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzm8s\" (UniqueName: \"kubernetes.io/projected/514c415b-492e-4092-9889-c2835f58b761-kube-api-access-dzm8s\") pod \"payload-processing-e2e-worker-w1-86d620-74c5fd4b96-2zkm5\" (UID: \"514c415b-492e-4092-9889-c2835f58b761\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-86d620-74c5fd4b96-2zkm5"
Oct 30 20:37:20.950107 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:20.950072    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:20.973513 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:20.973485    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-86d620-fb8d7795f-mjmqg"
Oct 30 20:37:21.005134 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:21.005107    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzm8s\" (UniqueName: \"kubernetes.io/projected/514c415b-492e-4092-9889-c2835f58b761-kube-api-access-dzm8s\") pod \"payload-processing-e2e-worker-w1-86d620-74c5fd4b96-2zkm5\" (UID: \"514c415b-492e-4092-9889-c2835f58b761\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-86d620-74c5fd4b96-2zkm5"
Oct 30 20:37:21.005261 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:21.005184    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/514c415b-492e-4092-9889-c2835f58b761-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-86d620-74c5fd4b96-2zkm5\" (UID: \"514c415b-492e-4092-9889-c2835f58b761\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-86d620-74c5fd4b96-2zkm5"
Oct 30 20:37:21.006075 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:21.006049    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/514c415b-492e-4092-9889-c2835f58b761-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-86d620-74c5fd4b96-2zkm5\" (UID: \"514c415b-492e-4092-9889-c2835f58b761\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-86d620-74c5fd4b96-2zkm5"
Oct 30 20:37:21.064464 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:21.064380    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzm8s\" (UniqueName: \"kubernetes.io/projected/514c415b-492e-4092-9889-c2835f58b761-kube-api-access-dzm8s\") pod \"payload-processing-e2e-worker-w1-86d620-74c5fd4b96-2zkm5\" (UID: \"514c415b-492e-4092-9889-c2835f58b761\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-86d620-74c5fd4b96-2zkm5"
Oct 30 20:37:21.118790 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:21.118740    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-86d620-74c5fd4b96-2zkm5"
Oct 30 20:37:21.151077 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:21.151047    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:21.355349 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:21.355315    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-86d620-fb8d7795f-mjmqg"]
Oct 30 20:37:21.358415 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:37:21.358391    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod904e938e_ad13_4460_9ab6_b1ce6ed3c1d6.slice/crio-142f4444ab38adf16c1d2578b6934b10a850d236f322845a2fc85277d904b099 WatchSource:0}: Error finding container 142f4444ab38adf16c1d2578b6934b10a850d236f322845a2fc85277d904b099: Status 404 returned error can't find the container with id 142f4444ab38adf16c1d2578b6934b10a850d236f322845a2fc85277d904b099
Oct 30 20:37:21.360284 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:21.360266    5071 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:37:21.475071 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:21.475033    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-86d620-fb8d7795f-mjmqg" event={"ID":"904e938e-ad13-4460-9ab6-b1ce6ed3c1d6","Type":"ContainerStarted","Data":"142f4444ab38adf16c1d2578b6934b10a850d236f322845a2fc85277d904b099"}
Oct 30 20:37:21.494099 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:21.494081    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-86d620-74c5fd4b96-2zkm5"]
Oct 30 20:37:21.496849 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:37:21.496798    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod514c415b_492e_4092_9889_c2835f58b761.slice/crio-ae95b8ac112aee23df6a129ea087ff9439b7a7d200c35bd0e201b0d8b7a590ce WatchSource:0}: Error finding container ae95b8ac112aee23df6a129ea087ff9439b7a7d200c35bd0e201b0d8b7a590ce: Status 404 returned error can't find the container with id ae95b8ac112aee23df6a129ea087ff9439b7a7d200c35bd0e201b0d8b7a590ce
Oct 30 20:37:21.846577 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:21.846534    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:22.019616 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:22.019584    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:22.188928 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:22.188839    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:22.484487 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:22.484398    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-86d620-74c5fd4b96-2zkm5" event={"ID":"514c415b-492e-4092-9889-c2835f58b761","Type":"ContainerStarted","Data":"ae95b8ac112aee23df6a129ea087ff9439b7a7d200c35bd0e201b0d8b7a590ce"}
Oct 30 20:37:22.736656 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:22.736568    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:22.953552 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:22.953513    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:24.042608 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:24.042567    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:24.980243 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:24.980199    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:25.374974 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:25.374934    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:25.501804 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:25.501764    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-86d620-fb8d7795f-mjmqg" event={"ID":"904e938e-ad13-4460-9ab6-b1ce6ed3c1d6","Type":"ContainerStarted","Data":"4c9dfe7e186d197b98d166719328a4df91bcb0319732c03a60124dcc9c986061"}
Oct 30 20:37:25.502002 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:25.501862    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-86d620-fb8d7795f-mjmqg"
Oct 30 20:37:25.503537 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:25.503515    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-86d620-74c5fd4b96-2zkm5" event={"ID":"514c415b-492e-4092-9889-c2835f58b761","Type":"ContainerStarted","Data":"1df1e4d8ec1594c9a65542c5944f756af17a466de11a134503ed67b3f99765a4"}
Oct 30 20:37:25.503645 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:25.503621    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-86d620-74c5fd4b96-2zkm5"
Oct 30 20:37:25.538473 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:25.538437    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:25.623979 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:25.623921    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-86d620-fb8d7795f-mjmqg" podStartSLOduration=2.566045464 podStartE2EDuration="5.62390093s" totalImagesPullingTime="3.057855466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:25.619687798 +0000 UTC m=+657.536709336" watchObservedRunningTime="2026-10-30 20:37:25.62390093 +0000 UTC m=+657.540922467"
Oct 30 20:37:25.738158 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:25.738063    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-86d620-74c5fd4b96-2zkm5" podStartSLOduration=2.786547312 podStartE2EDuration="5.738048754s" totalImagesPullingTime="2.951501442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:25.737681274 +0000 UTC m=+657.654702810" watchObservedRunningTime="2026-10-30 20:37:25.738048754 +0000 UTC m=+657.655070289"
Oct 30 20:37:25.744444 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:25.744415    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-ee3fad-69f6cc779f-plr9b"]
Oct 30 20:37:25.748383 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:25.748361    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee3fad-69f6cc779f-plr9b"
Oct 30 20:37:25.802095 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:25.802059    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-ee3fad\""
Oct 30 20:37:25.802095 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:25.802066    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-ee3fad-dockercfg-4vjtt\""
Oct 30 20:37:25.809066 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:25.809015    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-ee3fad-69f6cc779f-plr9b"]
Oct 30 20:37:25.871242 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:25.871036    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwt9q\" (UniqueName: \"kubernetes.io/projected/56a15d9e-6de3-47ab-8045-4a1c3618bc6a-kube-api-access-gwt9q\") pod \"payload-pre-processing-e2e-worker-w6-ee3fad-69f6cc779f-plr9b\" (UID: \"56a15d9e-6de3-47ab-8045-4a1c3618bc6a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee3fad-69f6cc779f-plr9b"
Oct 30 20:37:25.871242 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:25.871108    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56a15d9e-6de3-47ab-8045-4a1c3618bc6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-ee3fad-69f6cc779f-plr9b\" (UID: \"56a15d9e-6de3-47ab-8045-4a1c3618bc6a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee3fad-69f6cc779f-plr9b"
Oct 30 20:37:25.909230 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:25.909197    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-ee3fad-67dffbb55d-vjzhn"]
Oct 30 20:37:25.913012 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:25.912994    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-ee3fad-67dffbb55d-vjzhn"
Oct 30 20:37:25.971738 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:25.971700    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwt9q\" (UniqueName: \"kubernetes.io/projected/56a15d9e-6de3-47ab-8045-4a1c3618bc6a-kube-api-access-gwt9q\") pod \"payload-pre-processing-e2e-worker-w6-ee3fad-69f6cc779f-plr9b\" (UID: \"56a15d9e-6de3-47ab-8045-4a1c3618bc6a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee3fad-69f6cc779f-plr9b"
Oct 30 20:37:25.971973 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:25.971781    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20a5960c-1432-48a8-8b82-730d6973647d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-ee3fad-67dffbb55d-vjzhn\" (UID: \"20a5960c-1432-48a8-8b82-730d6973647d\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-ee3fad-67dffbb55d-vjzhn"
Oct 30 20:37:25.971973 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:25.971902    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56a15d9e-6de3-47ab-8045-4a1c3618bc6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-ee3fad-69f6cc779f-plr9b\" (UID: \"56a15d9e-6de3-47ab-8045-4a1c3618bc6a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee3fad-69f6cc779f-plr9b"
Oct 30 20:37:25.972108 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:25.972015    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbwhx\" (UniqueName: \"kubernetes.io/projected/20a5960c-1432-48a8-8b82-730d6973647d-kube-api-access-hbwhx\") pod \"payload-processing-e2e-worker-w6-ee3fad-67dffbb55d-vjzhn\" (UID: \"20a5960c-1432-48a8-8b82-730d6973647d\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-ee3fad-67dffbb55d-vjzhn"
Oct 30 20:37:25.972408 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:25.972373    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-ee3fad-67dffbb55d-vjzhn"]
Oct 30 20:37:25.972754 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:25.972731    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56a15d9e-6de3-47ab-8045-4a1c3618bc6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-ee3fad-69f6cc779f-plr9b\" (UID: \"56a15d9e-6de3-47ab-8045-4a1c3618bc6a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee3fad-69f6cc779f-plr9b"
Oct 30 20:37:26.032902 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:26.032814    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwt9q\" (UniqueName: \"kubernetes.io/projected/56a15d9e-6de3-47ab-8045-4a1c3618bc6a-kube-api-access-gwt9q\") pod \"payload-pre-processing-e2e-worker-w6-ee3fad-69f6cc779f-plr9b\" (UID: \"56a15d9e-6de3-47ab-8045-4a1c3618bc6a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee3fad-69f6cc779f-plr9b"
Oct 30 20:37:26.061858 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:26.061802    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee3fad-69f6cc779f-plr9b"
Oct 30 20:37:26.073294 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:26.073267    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbwhx\" (UniqueName: \"kubernetes.io/projected/20a5960c-1432-48a8-8b82-730d6973647d-kube-api-access-hbwhx\") pod \"payload-processing-e2e-worker-w6-ee3fad-67dffbb55d-vjzhn\" (UID: \"20a5960c-1432-48a8-8b82-730d6973647d\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-ee3fad-67dffbb55d-vjzhn"
Oct 30 20:37:26.073404 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:26.073361    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20a5960c-1432-48a8-8b82-730d6973647d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-ee3fad-67dffbb55d-vjzhn\" (UID: \"20a5960c-1432-48a8-8b82-730d6973647d\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-ee3fad-67dffbb55d-vjzhn"
Oct 30 20:37:26.074084 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:26.074063    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20a5960c-1432-48a8-8b82-730d6973647d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-ee3fad-67dffbb55d-vjzhn\" (UID: \"20a5960c-1432-48a8-8b82-730d6973647d\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-ee3fad-67dffbb55d-vjzhn"
Oct 30 20:37:26.134785 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:26.134748    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbwhx\" (UniqueName: \"kubernetes.io/projected/20a5960c-1432-48a8-8b82-730d6973647d-kube-api-access-hbwhx\") pod \"payload-processing-e2e-worker-w6-ee3fad-67dffbb55d-vjzhn\" (UID: \"20a5960c-1432-48a8-8b82-730d6973647d\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-ee3fad-67dffbb55d-vjzhn"
Oct 30 20:37:26.227457 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:26.227421    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-ee3fad-67dffbb55d-vjzhn"
Oct 30 20:37:26.437655 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:26.437632    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-ee3fad-69f6cc779f-plr9b"]
Oct 30 20:37:26.439306 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:37:26.439275    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a15d9e_6de3_47ab_8045_4a1c3618bc6a.slice/crio-b3fbb288b6d513079fe05f4f80f81f833222e6420f351eaa5c774e519682c7be WatchSource:0}: Error finding container b3fbb288b6d513079fe05f4f80f81f833222e6420f351eaa5c774e519682c7be: Status 404 returned error can't find the container with id b3fbb288b6d513079fe05f4f80f81f833222e6420f351eaa5c774e519682c7be
Oct 30 20:37:26.510572 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:26.510507    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee3fad-69f6cc779f-plr9b" event={"ID":"56a15d9e-6de3-47ab-8045-4a1c3618bc6a","Type":"ContainerStarted","Data":"b3fbb288b6d513079fe05f4f80f81f833222e6420f351eaa5c774e519682c7be"}
Oct 30 20:37:26.600868 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:26.600839    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-ee3fad-67dffbb55d-vjzhn"]
Oct 30 20:37:26.603551 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:37:26.603522    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a5960c_1432_48a8_8b82_730d6973647d.slice/crio-e9347ed3be2a07177c2ab66d7a24986817e894a33b3fc43edca7910c7d9a696d WatchSource:0}: Error finding container e9347ed3be2a07177c2ab66d7a24986817e894a33b3fc43edca7910c7d9a696d: Status 404 returned error can't find the container with id e9347ed3be2a07177c2ab66d7a24986817e894a33b3fc43edca7910c7d9a696d
Oct 30 20:37:27.208553 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:27.208518    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp"]
Oct 30 20:37:27.212456 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:27.212435    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp"
Oct 30 20:37:27.287217 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:27.287178    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84lq6\" (UniqueName: \"kubernetes.io/projected/b31f6255-ed4b-45af-9a92-e2de6f5ee723-kube-api-access-84lq6\") pod \"maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp\" (UID: \"b31f6255-ed4b-45af-9a92-e2de6f5ee723\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp"
Oct 30 20:37:27.287388 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:27.287339    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b31f6255-ed4b-45af-9a92-e2de6f5ee723-metrics-certs\") pod \"maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp\" (UID: \"b31f6255-ed4b-45af-9a92-e2de6f5ee723\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp"
Oct 30 20:37:27.287448 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:27.287422    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b31f6255-ed4b-45af-9a92-e2de6f5ee723-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp\" (UID: \"b31f6255-ed4b-45af-9a92-e2de6f5ee723\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp"
Oct 30 20:37:27.287565 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:27.287544    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b31f6255-ed4b-45af-9a92-e2de6f5ee723-maas-api-tls\") pod \"maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp\" (UID: \"b31f6255-ed4b-45af-9a92-e2de6f5ee723\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp"
Oct 30 20:37:27.318063 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:27.318029    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-f6ebf5-serving-cert\""
Oct 30 20:37:27.318214 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:27.318103    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:37:27.318281 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:27.318265    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 20:37:27.323181 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:27.318755    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rswnw\""
Oct 30 20:37:27.323181 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:27.318768    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:37:27.323181 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:27.322710    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp"]
Oct 30 20:37:27.388537 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:27.388506    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b31f6255-ed4b-45af-9a92-e2de6f5ee723-maas-api-tls\") pod \"maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp\" (UID: \"b31f6255-ed4b-45af-9a92-e2de6f5ee723\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp"
Oct 30 20:37:27.388687 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:27.388548    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84lq6\" (UniqueName: \"kubernetes.io/projected/b31f6255-ed4b-45af-9a92-e2de6f5ee723-kube-api-access-84lq6\") pod \"maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp\" (UID: \"b31f6255-ed4b-45af-9a92-e2de6f5ee723\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp"
Oct 30 20:37:27.388687 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:27.388614    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b31f6255-ed4b-45af-9a92-e2de6f5ee723-metrics-certs\") pod \"maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp\" (UID: \"b31f6255-ed4b-45af-9a92-e2de6f5ee723\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp"
Oct 30 20:37:27.388687 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:27.388646    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b31f6255-ed4b-45af-9a92-e2de6f5ee723-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp\" (UID: \"b31f6255-ed4b-45af-9a92-e2de6f5ee723\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp"
Oct 30 20:37:27.388687 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:37:27.388653    5071 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-f6ebf5-serving-cert: secret "maas-api-e2e-worker-w3-f6ebf5-serving-cert" not found
Oct 30 20:37:27.388861 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:37:27.388729    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b31f6255-ed4b-45af-9a92-e2de6f5ee723-maas-api-tls podName:b31f6255-ed4b-45af-9a92-e2de6f5ee723 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:27.888708999 +0000 UTC m=+659.805730527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b31f6255-ed4b-45af-9a92-e2de6f5ee723-maas-api-tls") pod "maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp" (UID: "b31f6255-ed4b-45af-9a92-e2de6f5ee723") : secret "maas-api-e2e-worker-w3-f6ebf5-serving-cert" not found
Oct 30 20:37:27.389486 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:27.389416    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b31f6255-ed4b-45af-9a92-e2de6f5ee723-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp\" (UID: \"b31f6255-ed4b-45af-9a92-e2de6f5ee723\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp"
Oct 30 20:37:27.391256 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:27.391233    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b31f6255-ed4b-45af-9a92-e2de6f5ee723-metrics-certs\") pod \"maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp\" (UID: \"b31f6255-ed4b-45af-9a92-e2de6f5ee723\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp"
Oct 30 20:37:27.451330 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:27.451299    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84lq6\" (UniqueName: \"kubernetes.io/projected/b31f6255-ed4b-45af-9a92-e2de6f5ee723-kube-api-access-84lq6\") pod \"maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp\" (UID: \"b31f6255-ed4b-45af-9a92-e2de6f5ee723\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp"
Oct 30 20:37:27.509044 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:27.508952    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:27.517428 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:27.517396    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee3fad-69f6cc779f-plr9b" event={"ID":"56a15d9e-6de3-47ab-8045-4a1c3618bc6a","Type":"ContainerStarted","Data":"14f626459dd48264a397c532e3c63cdf0dcbf06db5bcf38fd73177aa2b9eff50"}
Oct 30 20:37:27.517586 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:27.517565    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee3fad-69f6cc779f-plr9b"
Oct 30 20:37:27.519133 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:27.519108    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-ee3fad-67dffbb55d-vjzhn" event={"ID":"20a5960c-1432-48a8-8b82-730d6973647d","Type":"ContainerStarted","Data":"2dd06967f87a7f0bb535f1b3b6f992a79f9cf637fde6bbe8d1089c6cb4bafe1e"}
Oct 30 20:37:27.519245 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:27.519146    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-ee3fad-67dffbb55d-vjzhn" event={"ID":"20a5960c-1432-48a8-8b82-730d6973647d","Type":"ContainerStarted","Data":"e9347ed3be2a07177c2ab66d7a24986817e894a33b3fc43edca7910c7d9a696d"}
Oct 30 20:37:27.519245 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:27.519188    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-ee3fad-67dffbb55d-vjzhn"
Oct 30 20:37:27.634271 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:27.634206    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee3fad-69f6cc779f-plr9b" podStartSLOduration=2.634194393 podStartE2EDuration="2.634194393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:27.633518338 +0000 UTC m=+659.550539875" watchObservedRunningTime="2026-10-30 20:37:27.634194393 +0000 UTC m=+659.551215968"
Oct 30 20:37:27.720266 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:27.720235    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:27.892809 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:27.892775    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b31f6255-ed4b-45af-9a92-e2de6f5ee723-maas-api-tls\") pod \"maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp\" (UID: \"b31f6255-ed4b-45af-9a92-e2de6f5ee723\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp"
Oct 30 20:37:27.893101 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:27.893076    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:27.895322 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:27.895301    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b31f6255-ed4b-45af-9a92-e2de6f5ee723-maas-api-tls\") pod \"maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp\" (UID: \"b31f6255-ed4b-45af-9a92-e2de6f5ee723\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp"
Oct 30 20:37:28.126450 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:28.126416    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp"
Oct 30 20:37:28.184629 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:28.184604    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:28.344589 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:28.344552    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:28.382680 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:28.382635    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-ee3fad-67dffbb55d-vjzhn" podStartSLOduration=3.382619434 podStartE2EDuration="3.382619434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:27.763145149 +0000 UTC m=+659.680166712" watchObservedRunningTime="2026-10-30 20:37:28.382619434 +0000 UTC m=+660.299640973"
Oct 30 20:37:28.382877 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:28.382804    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f6ebf5-69984c68d7-ls9bm"]
Oct 30 20:37:28.387844 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:28.387801    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f6ebf5-69984c68d7-ls9bm"
Oct 30 20:37:28.471387 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:28.467695    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f6ebf5-69984c68d7-ls9bm"]
Oct 30 20:37:28.471963 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:28.470038    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-f6ebf5\""
Oct 30 20:37:28.472081 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:28.470202    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-f6ebf5-dockercfg-6d8dh\""
Oct 30 20:37:28.501427 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:28.501390    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bbebe9e-cd3b-49d7-8048-bde0a0faafa2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-f6ebf5-69984c68d7-ls9bm\" (UID: \"8bbebe9e-cd3b-49d7-8048-bde0a0faafa2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f6ebf5-69984c68d7-ls9bm"
Oct 30 20:37:28.501427 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:28.501441    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg45p\" (UniqueName: \"kubernetes.io/projected/8bbebe9e-cd3b-49d7-8048-bde0a0faafa2-kube-api-access-gg45p\") pod \"payload-pre-processing-e2e-worker-w3-f6ebf5-69984c68d7-ls9bm\" (UID: \"8bbebe9e-cd3b-49d7-8048-bde0a0faafa2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f6ebf5-69984c68d7-ls9bm"
Oct 30 20:37:28.523639 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:28.523568    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp"]
Oct 30 20:37:28.560900 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:37:28.560871    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb31f6255_ed4b_45af_9a92_e2de6f5ee723.slice/crio-fa15467f81a152cd0e7d1fe162d06a13eafab64ee8c4e69152a8d153236fbc37 WatchSource:0}: Error finding container fa15467f81a152cd0e7d1fe162d06a13eafab64ee8c4e69152a8d153236fbc37: Status 404 returned error can't find the container with id fa15467f81a152cd0e7d1fe162d06a13eafab64ee8c4e69152a8d153236fbc37
Oct 30 20:37:28.602476 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:28.602445    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bbebe9e-cd3b-49d7-8048-bde0a0faafa2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-f6ebf5-69984c68d7-ls9bm\" (UID: \"8bbebe9e-cd3b-49d7-8048-bde0a0faafa2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f6ebf5-69984c68d7-ls9bm"
Oct 30 20:37:28.602627 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:28.602484    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg45p\" (UniqueName: \"kubernetes.io/projected/8bbebe9e-cd3b-49d7-8048-bde0a0faafa2-kube-api-access-gg45p\") pod \"payload-pre-processing-e2e-worker-w3-f6ebf5-69984c68d7-ls9bm\" (UID: \"8bbebe9e-cd3b-49d7-8048-bde0a0faafa2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f6ebf5-69984c68d7-ls9bm"
Oct 30 20:37:28.603174 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:28.603156    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bbebe9e-cd3b-49d7-8048-bde0a0faafa2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-f6ebf5-69984c68d7-ls9bm\" (UID: \"8bbebe9e-cd3b-49d7-8048-bde0a0faafa2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f6ebf5-69984c68d7-ls9bm"
Oct 30 20:37:28.670025 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:28.670000    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg45p\" (UniqueName: \"kubernetes.io/projected/8bbebe9e-cd3b-49d7-8048-bde0a0faafa2-kube-api-access-gg45p\") pod \"payload-pre-processing-e2e-worker-w3-f6ebf5-69984c68d7-ls9bm\" (UID: \"8bbebe9e-cd3b-49d7-8048-bde0a0faafa2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f6ebf5-69984c68d7-ls9bm"
Oct 30 20:37:28.701002 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:28.700974    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f6ebf5-69984c68d7-ls9bm"
Oct 30 20:37:29.073619 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:29.073594    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f6ebf5-69984c68d7-ls9bm"]
Oct 30 20:37:29.075581 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:37:29.075550    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bbebe9e_cd3b_49d7_8048_bde0a0faafa2.slice/crio-0fd1ae2a5c363e3a41f924caea130373d3847170ac7bed4f49a203fe5eeee660 WatchSource:0}: Error finding container 0fd1ae2a5c363e3a41f924caea130373d3847170ac7bed4f49a203fe5eeee660: Status 404 returned error can't find the container with id 0fd1ae2a5c363e3a41f924caea130373d3847170ac7bed4f49a203fe5eeee660
Oct 30 20:37:29.535921 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:29.535808    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f6ebf5-69984c68d7-ls9bm" event={"ID":"8bbebe9e-cd3b-49d7-8048-bde0a0faafa2","Type":"ContainerStarted","Data":"934149a7dc3af11fe6a99e9e2eba31230ffc4ef52bc894929cf10393cfe256b7"}
Oct 30 20:37:29.536410 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:29.536066    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f6ebf5-69984c68d7-ls9bm"
Oct 30 20:37:29.536410 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:29.536115    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f6ebf5-69984c68d7-ls9bm" event={"ID":"8bbebe9e-cd3b-49d7-8048-bde0a0faafa2","Type":"ContainerStarted","Data":"0fd1ae2a5c363e3a41f924caea130373d3847170ac7bed4f49a203fe5eeee660"}
Oct 30 20:37:29.537645 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:29.537618    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp" event={"ID":"b31f6255-ed4b-45af-9a92-e2de6f5ee723","Type":"ContainerStarted","Data":"fa15467f81a152cd0e7d1fe162d06a13eafab64ee8c4e69152a8d153236fbc37"}
Oct 30 20:37:29.657437 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:29.657384    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f6ebf5-69984c68d7-ls9bm" podStartSLOduration=1.657370905 podStartE2EDuration="1.657370905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:29.656257426 +0000 UTC m=+661.573278963" watchObservedRunningTime="2026-10-30 20:37:29.657370905 +0000 UTC m=+661.574392468"
Oct 30 20:37:30.375166 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:30.375131    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:37:30.784725 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:30.784693    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:30.967553 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:30.967507    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:31.098468 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:31.098433    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-395022-5d767b7967-6nf69"]
Oct 30 20:37:31.102273 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:31.102256    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-395022-5d767b7967-6nf69"
Oct 30 20:37:31.159877 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:31.159845    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-395022\""
Oct 30 20:37:31.160092 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:31.159989    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-395022-dockercfg-cjhcf\""
Oct 30 20:37:31.163986 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:31.163291    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-395022-5d767b7967-6nf69"]
Oct 30 20:37:31.237465 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:31.235583    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxdlr\" (UniqueName: \"kubernetes.io/projected/6262b3d5-64e5-4e9c-a4b9-a08f25298572-kube-api-access-mxdlr\") pod \"payload-pre-processing-e2e-worker-w4-395022-5d767b7967-6nf69\" (UID: \"6262b3d5-64e5-4e9c-a4b9-a08f25298572\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-395022-5d767b7967-6nf69"
Oct 30 20:37:31.237465 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:31.235639    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6262b3d5-64e5-4e9c-a4b9-a08f25298572-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-395022-5d767b7967-6nf69\" (UID: \"6262b3d5-64e5-4e9c-a4b9-a08f25298572\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-395022-5d767b7967-6nf69"
Oct 30 20:37:31.337020 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:31.336988    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxdlr\" (UniqueName: \"kubernetes.io/projected/6262b3d5-64e5-4e9c-a4b9-a08f25298572-kube-api-access-mxdlr\") pod \"payload-pre-processing-e2e-worker-w4-395022-5d767b7967-6nf69\" (UID: \"6262b3d5-64e5-4e9c-a4b9-a08f25298572\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-395022-5d767b7967-6nf69"
Oct 30 20:37:31.337188 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:31.337029    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6262b3d5-64e5-4e9c-a4b9-a08f25298572-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-395022-5d767b7967-6nf69\" (UID: \"6262b3d5-64e5-4e9c-a4b9-a08f25298572\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-395022-5d767b7967-6nf69"
Oct 30 20:37:31.337733 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:31.337707    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6262b3d5-64e5-4e9c-a4b9-a08f25298572-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-395022-5d767b7967-6nf69\" (UID: \"6262b3d5-64e5-4e9c-a4b9-a08f25298572\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-395022-5d767b7967-6nf69"
Oct 30 20:37:31.399647 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:31.399574    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxdlr\" (UniqueName: \"kubernetes.io/projected/6262b3d5-64e5-4e9c-a4b9-a08f25298572-kube-api-access-mxdlr\") pod \"payload-pre-processing-e2e-worker-w4-395022-5d767b7967-6nf69\" (UID: \"6262b3d5-64e5-4e9c-a4b9-a08f25298572\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-395022-5d767b7967-6nf69"
Oct 30 20:37:31.416574 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:31.416547    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-395022-5d767b7967-6nf69"
Oct 30 20:37:31.553591 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:31.553554    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp" event={"ID":"b31f6255-ed4b-45af-9a92-e2de6f5ee723","Type":"ContainerStarted","Data":"ee31c7ba509840409ad4301374691ddbd2aa3ebe5c0b4755125130949d2bf4dd"}
Oct 30 20:37:31.553805 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:31.553713    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp"
Oct 30 20:37:31.673986 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:31.673871    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp" podStartSLOduration=2.461168335 podStartE2EDuration="4.673851395s" totalImagesPullingTime="2.21268306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:31.670617982 +0000 UTC m=+663.587639519" watchObservedRunningTime="2026-10-30 20:37:31.673851395 +0000 UTC m=+663.590872932"
Oct 30 20:37:31.800840 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:31.800791    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-395022-5d767b7967-6nf69"]
Oct 30 20:37:31.807025 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:37:31.806987    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6262b3d5_64e5_4e9c_a4b9_a08f25298572.slice/crio-2c7626ca7b1109b360c85002a901997ba79f43bf0f133e8aa475a15b4c49c399 WatchSource:0}: Error finding container 2c7626ca7b1109b360c85002a901997ba79f43bf0f133e8aa475a15b4c49c399: Status 404 returned error can't find the container with id 2c7626ca7b1109b360c85002a901997ba79f43bf0f133e8aa475a15b4c49c399
Oct 30 20:37:32.561083 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:32.561050    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-395022-5d767b7967-6nf69" event={"ID":"6262b3d5-64e5-4e9c-a4b9-a08f25298572","Type":"ContainerStarted","Data":"c1e97d01d1ac68576dadef9d6de7a2a1f2cd9431c29b0cd327e8c18463ceb380"}
Oct 30 20:37:32.561258 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:32.561138    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-395022-5d767b7967-6nf69" event={"ID":"6262b3d5-64e5-4e9c-a4b9-a08f25298572","Type":"ContainerStarted","Data":"2c7626ca7b1109b360c85002a901997ba79f43bf0f133e8aa475a15b4c49c399"}
Oct 30 20:37:32.561336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:32.561274    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-395022-5d767b7967-6nf69"
Oct 30 20:37:32.678734 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:32.678674    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-395022-5d767b7967-6nf69" podStartSLOduration=1.678658043 podStartE2EDuration="1.678658043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:32.676562982 +0000 UTC m=+664.593584519" watchObservedRunningTime="2026-10-30 20:37:32.678658043 +0000 UTC m=+664.595679578"
Oct 30 20:37:32.741014 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:32.740975    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:32.958629 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:32.958540    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:33.117089 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:33.117058    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:35.840889 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:35.840850    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:36.047447 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:36.047395    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:36.245798 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:36.245723    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-386743-5894c7467-txx44"]
Oct 30 20:37:36.249655 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:36.249640    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-386743-5894c7467-txx44"
Oct 30 20:37:36.307722 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:36.307684    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-386743-5894c7467-txx44"]
Oct 30 20:37:36.356496 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:36.356464    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-386743\""
Oct 30 20:37:36.356892 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:36.356857    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-386743-dockercfg-h9jpd\""
Oct 30 20:37:36.392892 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:36.392858    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1beb9127-eb1c-41cf-8217-52ec9d5d258c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-386743-5894c7467-txx44\" (UID: \"1beb9127-eb1c-41cf-8217-52ec9d5d258c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-386743-5894c7467-txx44"
Oct 30 20:37:36.393050 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:36.392994    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsj6k\" (UniqueName: \"kubernetes.io/projected/1beb9127-eb1c-41cf-8217-52ec9d5d258c-kube-api-access-zsj6k\") pod \"payload-pre-processing-e2e-worker-w5-386743-5894c7467-txx44\" (UID: \"1beb9127-eb1c-41cf-8217-52ec9d5d258c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-386743-5894c7467-txx44"
Oct 30 20:37:36.408540 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:36.408510    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-386743-68d76cd7d6-xczl5"]
Oct 30 20:37:36.412609 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:36.412586    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-386743-68d76cd7d6-xczl5"
Oct 30 20:37:36.455364 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:36.455326    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-386743-68d76cd7d6-xczl5"]
Oct 30 20:37:36.494467 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:36.494437    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpnjg\" (UniqueName: \"kubernetes.io/projected/49cfe5da-09b4-4268-8938-4d6f10827d35-kube-api-access-gpnjg\") pod \"payload-processing-e2e-worker-w5-386743-68d76cd7d6-xczl5\" (UID: \"49cfe5da-09b4-4268-8938-4d6f10827d35\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-386743-68d76cd7d6-xczl5"
Oct 30 20:37:36.494630 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:36.494476    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49cfe5da-09b4-4268-8938-4d6f10827d35-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-386743-68d76cd7d6-xczl5\" (UID: \"49cfe5da-09b4-4268-8938-4d6f10827d35\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-386743-68d76cd7d6-xczl5"
Oct 30 20:37:36.494630 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:36.494502    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1beb9127-eb1c-41cf-8217-52ec9d5d258c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-386743-5894c7467-txx44\" (UID: \"1beb9127-eb1c-41cf-8217-52ec9d5d258c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-386743-5894c7467-txx44"
Oct 30 20:37:36.494751 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:36.494730    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsj6k\" (UniqueName: \"kubernetes.io/projected/1beb9127-eb1c-41cf-8217-52ec9d5d258c-kube-api-access-zsj6k\") pod \"payload-pre-processing-e2e-worker-w5-386743-5894c7467-txx44\" (UID: \"1beb9127-eb1c-41cf-8217-52ec9d5d258c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-386743-5894c7467-txx44"
Oct 30 20:37:36.495173 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:36.495154    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1beb9127-eb1c-41cf-8217-52ec9d5d258c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-386743-5894c7467-txx44\" (UID: \"1beb9127-eb1c-41cf-8217-52ec9d5d258c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-386743-5894c7467-txx44"
Oct 30 20:37:36.514066 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:36.514015    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-86d620-74c5fd4b96-2zkm5"
Oct 30 20:37:36.514158 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:36.514084    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-86d620-fb8d7795f-mjmqg"
Oct 30 20:37:36.561769 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:36.561734    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsj6k\" (UniqueName: \"kubernetes.io/projected/1beb9127-eb1c-41cf-8217-52ec9d5d258c-kube-api-access-zsj6k\") pod \"payload-pre-processing-e2e-worker-w5-386743-5894c7467-txx44\" (UID: \"1beb9127-eb1c-41cf-8217-52ec9d5d258c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-386743-5894c7467-txx44"
Oct 30 20:37:36.564441 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:36.564421    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-386743-5894c7467-txx44"
Oct 30 20:37:36.595573 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:36.595536    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpnjg\" (UniqueName: \"kubernetes.io/projected/49cfe5da-09b4-4268-8938-4d6f10827d35-kube-api-access-gpnjg\") pod \"payload-processing-e2e-worker-w5-386743-68d76cd7d6-xczl5\" (UID: \"49cfe5da-09b4-4268-8938-4d6f10827d35\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-386743-68d76cd7d6-xczl5"
Oct 30 20:37:36.595716 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:36.595591    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49cfe5da-09b4-4268-8938-4d6f10827d35-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-386743-68d76cd7d6-xczl5\" (UID: \"49cfe5da-09b4-4268-8938-4d6f10827d35\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-386743-68d76cd7d6-xczl5"
Oct 30 20:37:36.596366 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:36.596343    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49cfe5da-09b4-4268-8938-4d6f10827d35-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-386743-68d76cd7d6-xczl5\" (UID: \"49cfe5da-09b4-4268-8938-4d6f10827d35\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-386743-68d76cd7d6-xczl5"
Oct 30 20:37:36.657110 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:36.657076    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpnjg\" (UniqueName: \"kubernetes.io/projected/49cfe5da-09b4-4268-8938-4d6f10827d35-kube-api-access-gpnjg\") pod \"payload-processing-e2e-worker-w5-386743-68d76cd7d6-xczl5\" (UID: \"49cfe5da-09b4-4268-8938-4d6f10827d35\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-386743-68d76cd7d6-xczl5"
Oct 30 20:37:36.726351 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:36.726321    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-386743-68d76cd7d6-xczl5"
Oct 30 20:37:36.940630 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:36.940604    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-386743-5894c7467-txx44"]
Oct 30 20:37:36.942411 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:37:36.942383    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1beb9127_eb1c_41cf_8217_52ec9d5d258c.slice/crio-876d2eee5624cd47d577d5b813c23c2326957f9e60ad596c36f4d1eb0b84a4b9 WatchSource:0}: Error finding container 876d2eee5624cd47d577d5b813c23c2326957f9e60ad596c36f4d1eb0b84a4b9: Status 404 returned error can't find the container with id 876d2eee5624cd47d577d5b813c23c2326957f9e60ad596c36f4d1eb0b84a4b9
Oct 30 20:37:37.106503 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:37.106479    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-386743-68d76cd7d6-xczl5"]
Oct 30 20:37:37.107958 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:37:37.107931    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49cfe5da_09b4_4268_8938_4d6f10827d35.slice/crio-2cb49bf7349aff98f5d26dd120a2dc4b397136492190323ea64454d3d2a7225e WatchSource:0}: Error finding container 2cb49bf7349aff98f5d26dd120a2dc4b397136492190323ea64454d3d2a7225e: Status 404 returned error can't find the container with id 2cb49bf7349aff98f5d26dd120a2dc4b397136492190323ea64454d3d2a7225e
Oct 30 20:37:37.567957 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:37.567925    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp"
Oct 30 20:37:37.590988 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:37.590953    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-386743-68d76cd7d6-xczl5" event={"ID":"49cfe5da-09b4-4268-8938-4d6f10827d35","Type":"ContainerStarted","Data":"ff757a494de40c04f6d110d862a9df9f1f16e16e49ef8749006fb984c105dae0"}
Oct 30 20:37:37.591156 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:37.591015    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-386743-68d76cd7d6-xczl5" event={"ID":"49cfe5da-09b4-4268-8938-4d6f10827d35","Type":"ContainerStarted","Data":"2cb49bf7349aff98f5d26dd120a2dc4b397136492190323ea64454d3d2a7225e"}
Oct 30 20:37:37.591156 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:37.591049    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-386743-68d76cd7d6-xczl5"
Oct 30 20:37:37.592513 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:37.592488    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-386743-5894c7467-txx44" event={"ID":"1beb9127-eb1c-41cf-8217-52ec9d5d258c","Type":"ContainerStarted","Data":"567e07ffad1124868e7f487828ac33a2ea753211a0936fd5098ee30337e6d1db"}
Oct 30 20:37:37.592634 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:37.592525    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-386743-5894c7467-txx44" event={"ID":"1beb9127-eb1c-41cf-8217-52ec9d5d258c","Type":"ContainerStarted","Data":"876d2eee5624cd47d577d5b813c23c2326957f9e60ad596c36f4d1eb0b84a4b9"}
Oct 30 20:37:37.592634 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:37.592563    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-386743-5894c7467-txx44"
Oct 30 20:37:37.653123 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:37.653090    5071 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-149.us-west-1.compute.internal" podUID="9d9f66ca5c88277c8c76e3090f63988d" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Oct 30 20:37:37.804877 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:37.804788    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-386743-5894c7467-txx44" podStartSLOduration=1.804770201 podStartE2EDuration="1.804770201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:37.804453101 +0000 UTC m=+669.721474637" watchObservedRunningTime="2026-10-30 20:37:37.804770201 +0000 UTC m=+669.721791741"
Oct 30 20:37:37.932767 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:37.932633    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-386743-68d76cd7d6-xczl5" podStartSLOduration=1.9326124569999998 podStartE2EDuration="1.932612457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:37.929432223 +0000 UTC m=+669.846453760" watchObservedRunningTime="2026-10-30 20:37:37.932612457 +0000 UTC m=+669.849633995"
Oct 30 20:37:38.018628 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:38.018585    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:38.225866 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:38.225762    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:38.405297 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:38.405252    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:38.531277 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:38.531195    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee3fad-69f6cc779f-plr9b"
Oct 30 20:37:38.531461 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:38.531335    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-ee3fad-67dffbb55d-vjzhn"
Oct 30 20:37:39.715360 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:39.710123    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:37:40.547401 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:40.547375    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f6ebf5-69984c68d7-ls9bm"
Oct 30 20:37:40.916616 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:40.916522    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:41.154036 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:41.154002    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:41.303511 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:41.303473    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b32941-57bff4bc8c-qgdg8"]
Oct 30 20:37:41.307411 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:41.307389    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b32941-57bff4bc8c-qgdg8"
Oct 30 20:37:41.368128 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:41.368097    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b32941-57bff4bc8c-qgdg8"]
Oct 30 20:37:41.412011 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:41.411967    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-b32941-dockercfg-c8ltn\""
Oct 30 20:37:41.412207 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:41.412118    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-b32941\""
Oct 30 20:37:41.479495 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:41.479413    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f218d322-bfbb-4396-8152-5b1138f476e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b32941-57bff4bc8c-qgdg8\" (UID: \"f218d322-bfbb-4396-8152-5b1138f476e8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b32941-57bff4bc8c-qgdg8"
Oct 30 20:37:41.479691 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:41.479531    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdkl9\" (UniqueName: \"kubernetes.io/projected/f218d322-bfbb-4396-8152-5b1138f476e8-kube-api-access-rdkl9\") pod \"payload-pre-processing-e2e-worker-w0-b32941-57bff4bc8c-qgdg8\" (UID: \"f218d322-bfbb-4396-8152-5b1138f476e8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b32941-57bff4bc8c-qgdg8"
Oct 30 20:37:41.481119 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:41.481097    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b32941-66c49c7d84-pj6jk"]
Oct 30 20:37:41.484967 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:41.484949    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b32941-66c49c7d84-pj6jk"
Oct 30 20:37:41.500669 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:41.500642    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b32941-66c49c7d84-pj6jk"]
Oct 30 20:37:41.580930 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:41.580829    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgjhv\" (UniqueName: \"kubernetes.io/projected/6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f-kube-api-access-lgjhv\") pod \"payload-processing-e2e-worker-w0-b32941-66c49c7d84-pj6jk\" (UID: \"6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b32941-66c49c7d84-pj6jk"
Oct 30 20:37:41.580930 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:41.580875    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f218d322-bfbb-4396-8152-5b1138f476e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b32941-57bff4bc8c-qgdg8\" (UID: \"f218d322-bfbb-4396-8152-5b1138f476e8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b32941-57bff4bc8c-qgdg8"
Oct 30 20:37:41.581171 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:41.580976    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdkl9\" (UniqueName: \"kubernetes.io/projected/f218d322-bfbb-4396-8152-5b1138f476e8-kube-api-access-rdkl9\") pod \"payload-pre-processing-e2e-worker-w0-b32941-57bff4bc8c-qgdg8\" (UID: \"f218d322-bfbb-4396-8152-5b1138f476e8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b32941-57bff4bc8c-qgdg8"
Oct 30 20:37:41.581171 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:41.581086    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b32941-66c49c7d84-pj6jk\" (UID: \"6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b32941-66c49c7d84-pj6jk"
Oct 30 20:37:41.581700 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:41.581679    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f218d322-bfbb-4396-8152-5b1138f476e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b32941-57bff4bc8c-qgdg8\" (UID: \"f218d322-bfbb-4396-8152-5b1138f476e8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b32941-57bff4bc8c-qgdg8"
Oct 30 20:37:41.659792 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:41.659758    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdkl9\" (UniqueName: \"kubernetes.io/projected/f218d322-bfbb-4396-8152-5b1138f476e8-kube-api-access-rdkl9\") pod \"payload-pre-processing-e2e-worker-w0-b32941-57bff4bc8c-qgdg8\" (UID: \"f218d322-bfbb-4396-8152-5b1138f476e8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b32941-57bff4bc8c-qgdg8"
Oct 30 20:37:41.681951 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:41.681915    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgjhv\" (UniqueName: \"kubernetes.io/projected/6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f-kube-api-access-lgjhv\") pod \"payload-processing-e2e-worker-w0-b32941-66c49c7d84-pj6jk\" (UID: \"6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b32941-66c49c7d84-pj6jk"
Oct 30 20:37:41.682130 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:41.682014    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b32941-66c49c7d84-pj6jk\" (UID: \"6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b32941-66c49c7d84-pj6jk"
Oct 30 20:37:41.682670 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:41.682645    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b32941-66c49c7d84-pj6jk\" (UID: \"6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b32941-66c49c7d84-pj6jk"
Oct 30 20:37:41.761200 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:41.761138    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgjhv\" (UniqueName: \"kubernetes.io/projected/6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f-kube-api-access-lgjhv\") pod \"payload-processing-e2e-worker-w0-b32941-66c49c7d84-pj6jk\" (UID: \"6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b32941-66c49c7d84-pj6jk"
Oct 30 20:37:41.799773 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:41.799735    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b32941-66c49c7d84-pj6jk"
Oct 30 20:37:41.921078 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:41.921050    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b32941-57bff4bc8c-qgdg8"
Oct 30 20:37:42.209131 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:42.209049    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b32941-66c49c7d84-pj6jk"]
Oct 30 20:37:42.212932 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:37:42.212885    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6908b42f_ec2a_4943_8a1f_fd8b0ef07e8f.slice/crio-6d5b491c242948419fb964331604aea47009441e786d2e0d4acf5fcbd47bc247 WatchSource:0}: Error finding container 6d5b491c242948419fb964331604aea47009441e786d2e0d4acf5fcbd47bc247: Status 404 returned error can't find the container with id 6d5b491c242948419fb964331604aea47009441e786d2e0d4acf5fcbd47bc247
Oct 30 20:37:42.356421 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:42.348802    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b32941-57bff4bc8c-qgdg8"]
Oct 30 20:37:42.357623 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:37:42.357598    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf218d322_bfbb_4396_8152_5b1138f476e8.slice/crio-56fff16e7121d9690657f791f6d25ca64bbf1d73b04ca8b4dd9a2ba557cc388a WatchSource:0}: Error finding container 56fff16e7121d9690657f791f6d25ca64bbf1d73b04ca8b4dd9a2ba557cc388a: Status 404 returned error can't find the container with id 56fff16e7121d9690657f791f6d25ca64bbf1d73b04ca8b4dd9a2ba557cc388a
Oct 30 20:37:42.621378 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:42.621344    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b32941-57bff4bc8c-qgdg8" event={"ID":"f218d322-bfbb-4396-8152-5b1138f476e8","Type":"ContainerStarted","Data":"52f1f49ab17276a0da6c8db5706a859d4b7d7112b3037bff95cb673509f7aa25"}
Oct 30 20:37:42.621561 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:42.621409    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b32941-57bff4bc8c-qgdg8" event={"ID":"f218d322-bfbb-4396-8152-5b1138f476e8","Type":"ContainerStarted","Data":"56fff16e7121d9690657f791f6d25ca64bbf1d73b04ca8b4dd9a2ba557cc388a"}
Oct 30 20:37:42.621561 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:42.621442    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b32941-57bff4bc8c-qgdg8"
Oct 30 20:37:42.622946 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:42.622921    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b32941-66c49c7d84-pj6jk" event={"ID":"6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f","Type":"ContainerStarted","Data":"47aa1564ddf759b7df0a648ac36dc75f1160382b129c0a3eece71243f10f06ca"}
Oct 30 20:37:42.623047 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:42.622971    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b32941-66c49c7d84-pj6jk" event={"ID":"6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f","Type":"ContainerStarted","Data":"6d5b491c242948419fb964331604aea47009441e786d2e0d4acf5fcbd47bc247"}
Oct 30 20:37:42.623086 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:42.623052    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-b32941-66c49c7d84-pj6jk"
Oct 30 20:37:42.868615 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:42.867259    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b32941-57bff4bc8c-qgdg8" podStartSLOduration=1.867237821 podStartE2EDuration="1.867237821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:42.749107002 +0000 UTC m=+674.666128539" watchObservedRunningTime="2026-10-30 20:37:42.867237821 +0000 UTC m=+674.784259356"
Oct 30 20:37:42.869870 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:42.869800    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-b32941-66c49c7d84-pj6jk" podStartSLOduration=1.869780461 podStartE2EDuration="1.869780461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:42.865925589 +0000 UTC m=+674.782947117" watchObservedRunningTime="2026-10-30 20:37:42.869780461 +0000 UTC m=+674.786801999"
Oct 30 20:37:43.259250 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:43.259212    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:43.488704 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:43.487113    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:43.569730 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:43.569700    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-395022-5d767b7967-6nf69"
Oct 30 20:37:43.654855 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:43.652209    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:46.259652 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:46.259611    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:46.411704 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:46.411673    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:46.624109 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:46.624080    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b43c84-7cc684fccf-k6nvt"]
Oct 30 20:37:46.628645 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:46.628616    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b43c84-7cc684fccf-k6nvt"
Oct 30 20:37:46.734414 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:46.734379    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-b43c84-dockercfg-clphk\""
Oct 30 20:37:46.734859 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:46.734753    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-b43c84\""
Oct 30 20:37:46.744411 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:46.743370    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz6vq\" (UniqueName: \"kubernetes.io/projected/b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4-kube-api-access-dz6vq\") pod \"payload-pre-processing-e2e-worker-w2-b43c84-7cc684fccf-k6nvt\" (UID: \"b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b43c84-7cc684fccf-k6nvt"
Oct 30 20:37:46.744769 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:46.744745    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-b43c84-7cc684fccf-k6nvt\" (UID: \"b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b43c84-7cc684fccf-k6nvt"
Oct 30 20:37:46.747224 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:46.746178    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b43c84-7cc684fccf-k6nvt"]
Oct 30 20:37:46.773120 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:37:46.773082    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:37:46.786599 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:46.786572    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b43c84-b4788d4f4-tg275"]
Oct 30 20:37:46.790589 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:46.790569    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-b43c84-b4788d4f4-tg275"
Oct 30 20:37:46.836172 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:46.836086    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b43c84-b4788d4f4-tg275"]
Oct 30 20:37:46.845627 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:46.845594    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz6vq\" (UniqueName: \"kubernetes.io/projected/b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4-kube-api-access-dz6vq\") pod \"payload-pre-processing-e2e-worker-w2-b43c84-7cc684fccf-k6nvt\" (UID: \"b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b43c84-7cc684fccf-k6nvt"
Oct 30 20:37:46.845863 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:46.845738    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-b43c84-7cc684fccf-k6nvt\" (UID: \"b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b43c84-7cc684fccf-k6nvt"
Oct 30 20:37:46.846717 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:46.846667    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-b43c84-7cc684fccf-k6nvt\" (UID: \"b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b43c84-7cc684fccf-k6nvt"
Oct 30 20:37:46.909230 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:46.909144    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz6vq\" (UniqueName: \"kubernetes.io/projected/b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4-kube-api-access-dz6vq\") pod \"payload-pre-processing-e2e-worker-w2-b43c84-7cc684fccf-k6nvt\" (UID: \"b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b43c84-7cc684fccf-k6nvt"
Oct 30 20:37:46.946696 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:46.946650    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b43c84-7cc684fccf-k6nvt"
Oct 30 20:37:46.947286 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:46.947249    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dmp6\" (UniqueName: \"kubernetes.io/projected/78e27687-f62d-44a3-a67d-1b3397a45a00-kube-api-access-8dmp6\") pod \"payload-processing-e2e-worker-w2-b43c84-b4788d4f4-tg275\" (UID: \"78e27687-f62d-44a3-a67d-1b3397a45a00\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b43c84-b4788d4f4-tg275"
Oct 30 20:37:46.947388 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:46.947304    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78e27687-f62d-44a3-a67d-1b3397a45a00-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-b43c84-b4788d4f4-tg275\" (UID: \"78e27687-f62d-44a3-a67d-1b3397a45a00\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b43c84-b4788d4f4-tg275"
Oct 30 20:37:47.052097 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:47.048285    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dmp6\" (UniqueName: \"kubernetes.io/projected/78e27687-f62d-44a3-a67d-1b3397a45a00-kube-api-access-8dmp6\") pod \"payload-processing-e2e-worker-w2-b43c84-b4788d4f4-tg275\" (UID: \"78e27687-f62d-44a3-a67d-1b3397a45a00\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b43c84-b4788d4f4-tg275"
Oct 30 20:37:47.052097 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:47.048348    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78e27687-f62d-44a3-a67d-1b3397a45a00-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-b43c84-b4788d4f4-tg275\" (UID: \"78e27687-f62d-44a3-a67d-1b3397a45a00\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b43c84-b4788d4f4-tg275"
Oct 30 20:37:47.052097 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:47.049342    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78e27687-f62d-44a3-a67d-1b3397a45a00-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-b43c84-b4788d4f4-tg275\" (UID: \"78e27687-f62d-44a3-a67d-1b3397a45a00\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b43c84-b4788d4f4-tg275"
Oct 30 20:37:47.113159 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:47.113124    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dmp6\" (UniqueName: \"kubernetes.io/projected/78e27687-f62d-44a3-a67d-1b3397a45a00-kube-api-access-8dmp6\") pod \"payload-processing-e2e-worker-w2-b43c84-b4788d4f4-tg275\" (UID: \"78e27687-f62d-44a3-a67d-1b3397a45a00\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b43c84-b4788d4f4-tg275"
Oct 30 20:37:47.372781 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:47.372754    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b43c84-7cc684fccf-k6nvt"]
Oct 30 20:37:47.375091 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:37:47.375058    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0d1e19b_d2f0_4c2d_bcdc_9da486200fd4.slice/crio-b5c28b9d7ed5af24cd2d9aa6bb705b4436c334202daf824cdcdbf92c7fa45832 WatchSource:0}: Error finding container b5c28b9d7ed5af24cd2d9aa6bb705b4436c334202daf824cdcdbf92c7fa45832: Status 404 returned error can't find the container with id b5c28b9d7ed5af24cd2d9aa6bb705b4436c334202daf824cdcdbf92c7fa45832
Oct 30 20:37:47.407268 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:47.407238    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-b43c84-b4788d4f4-tg275"
Oct 30 20:37:47.676722 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:47.676632    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b43c84-7cc684fccf-k6nvt" event={"ID":"b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4","Type":"ContainerStarted","Data":"294e41bfc1e3e2006f09429290fd18ce63ce1b4c6e6e264e0e61edceabc1bf0b"}
Oct 30 20:37:47.676722 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:47.676690    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b43c84-7cc684fccf-k6nvt" event={"ID":"b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4","Type":"ContainerStarted","Data":"b5c28b9d7ed5af24cd2d9aa6bb705b4436c334202daf824cdcdbf92c7fa45832"}
Oct 30 20:37:47.677006 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:47.676854    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b43c84-7cc684fccf-k6nvt"
Oct 30 20:37:47.790558 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:47.790534    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b43c84-b4788d4f4-tg275"]
Oct 30 20:37:47.792640 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:37:47.792605    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e27687_f62d_44a3_a67d_1b3397a45a00.slice/crio-ebf7c663841f67df646a83f3322760a1fc9a5d2d1549ac3fbd2f75ef6fb3e3f9 WatchSource:0}: Error finding container ebf7c663841f67df646a83f3322760a1fc9a5d2d1549ac3fbd2f75ef6fb3e3f9: Status 404 returned error can't find the container with id ebf7c663841f67df646a83f3322760a1fc9a5d2d1549ac3fbd2f75ef6fb3e3f9
Oct 30 20:37:47.797923 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:47.797871    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b43c84-7cc684fccf-k6nvt" podStartSLOduration=1.79785432 podStartE2EDuration="1.79785432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:47.796961765 +0000 UTC m=+679.713983302" watchObservedRunningTime="2026-10-30 20:37:47.79785432 +0000 UTC m=+679.714875857"
Oct 30 20:37:48.600977 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:48.600945    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-386743-5894c7467-txx44"
Oct 30 20:37:48.601765 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:48.601236    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-386743-68d76cd7d6-xczl5"
Oct 30 20:37:48.665728 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:48.665696    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:48.684979 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:48.684946    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-b43c84-b4788d4f4-tg275" event={"ID":"78e27687-f62d-44a3-a67d-1b3397a45a00","Type":"ContainerStarted","Data":"60c32c19ae9c622c638cb3375cd7801302d57f06d5733d5f6eb7cee17318635a"}
Oct 30 20:37:48.685132 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:48.685036    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-b43c84-b4788d4f4-tg275" event={"ID":"78e27687-f62d-44a3-a67d-1b3397a45a00","Type":"ContainerStarted","Data":"ebf7c663841f67df646a83f3322760a1fc9a5d2d1549ac3fbd2f75ef6fb3e3f9"}
Oct 30 20:37:48.951139 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:48.951016    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-b43c84-b4788d4f4-tg275" podStartSLOduration=2.950998228 podStartE2EDuration="2.950998228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:48.950031078 +0000 UTC m=+680.867052616" watchObservedRunningTime="2026-10-30 20:37:48.950998228 +0000 UTC m=+680.868019765"
Oct 30 20:37:49.691172 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:49.691142    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-b43c84-b4788d4f4-tg275"
Oct 30 20:37:50.697760 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:50.697725    5071 generic.go:356] "Generic (PLEG): container finished" podID="5863cbee-17a0-4b2e-a6e5-c46737b72491" containerID="fab3657d130ceced6ecc5f39b5a6707979abc9d0738657294ab14ba60df5fe37" exitCode=0
Oct 30 20:37:50.698290 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:50.697803    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d" event={"ID":"5863cbee-17a0-4b2e-a6e5-c46737b72491","Type":"ContainerDied","Data":"fab3657d130ceced6ecc5f39b5a6707979abc9d0738657294ab14ba60df5fe37"}
Oct 30 20:37:50.698353 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:50.698338    5071 scope.go:122] "RemoveContainer" containerID="fab3657d130ceced6ecc5f39b5a6707979abc9d0738657294ab14ba60df5fe37"
Oct 30 20:37:50.699566 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:50.699533    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:37:50.699675 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:50.699618    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:37:50.699675 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:50.699666    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:37:50.832305 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:50.821598    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:37:51.706352 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:51.706317    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d" event={"ID":"5863cbee-17a0-4b2e-a6e5-c46737b72491","Type":"ContainerStarted","Data":"cdd1b09017184a821daf6d83c1295ebbe916122d53662895a7f6aee4c6f9878a"}
Oct 30 20:37:52.020494 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:52.020414    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:37:52.020679 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:52.020660    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 20:37:52.020722 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:52.020705    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d" podUID="5863cbee-17a0-4b2e-a6e5-c46737b72491" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 20:37:53.021029 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:53.020992    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 20:37:53.021421 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:53.021044    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d" podUID="5863cbee-17a0-4b2e-a6e5-c46737b72491" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 20:37:53.661694 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:53.661661    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-b32941-66c49c7d84-pj6jk"
Oct 30 20:37:53.661907 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:53.661755    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b32941-57bff4bc8c-qgdg8"
Oct 30 20:37:53.695520 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:53.695487    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:53.918268 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:53.918175    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:54.021408 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:54.021376    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 20:37:54.021794 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:54.021433    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d" podUID="5863cbee-17a0-4b2e-a6e5-c46737b72491" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 20:37:54.083623 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:54.083585    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:37:55.020571 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:55.020532    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 20:37:55.020755 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:55.020594    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d" podUID="5863cbee-17a0-4b2e-a6e5-c46737b72491" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 20:37:56.021195 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:56.021158    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 20:37:56.021586 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:56.021219    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d" podUID="5863cbee-17a0-4b2e-a6e5-c46737b72491" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 20:37:57.025741 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:57.025712    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:37:57.026234 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:57.026006    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:37:57.026721 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:57.026706    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:37:58.689169 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:58.689137    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b43c84-7cc684fccf-k6nvt"
Oct 30 20:37:58.877471 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:37:58.877442    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:00.701283 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:00.701255    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-b43c84-b4788d4f4-tg275"
Oct 30 20:38:02.397961 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:02.397921    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:38:03.831452 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:03.831401    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:04.000353 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:04.000314    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:04.152434 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:04.152334    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:05.785867 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:05.785805    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:06.410918 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:06.410887    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:06.591580 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:06.591527    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:09.560840 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:09.560788    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:09.793668 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:09.793627    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:13.658894 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:13.658863    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:38:14.936904 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:14.936866    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:16.139140 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:16.139106    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:16.746877 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:38:16.746814    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:38:16.803179 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:16.803086    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:17.055894 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:17.055772    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:20.682713 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:20.682676    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:22.819389 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:22.819348    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:23.225941 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:23.225860    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:23.493385 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:23.493289    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:23.672757 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:23.672576    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:23.943235 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:23.943200    5071 generic.go:356] "Generic (PLEG): container finished" podID="99069007-e0f8-4092-8fd6-f27b0a119720" containerID="63d8251f784c6fab20cf2a5ba32c42a286a091cc3a95f12eff1c80c5ddef01db" exitCode=0
Oct 30 20:38:23.943734 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:23.943269    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6" event={"ID":"99069007-e0f8-4092-8fd6-f27b0a119720","Type":"ContainerDied","Data":"63d8251f784c6fab20cf2a5ba32c42a286a091cc3a95f12eff1c80c5ddef01db"}
Oct 30 20:38:23.943804 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:23.943746    5071 scope.go:122] "RemoveContainer" containerID="63d8251f784c6fab20cf2a5ba32c42a286a091cc3a95f12eff1c80c5ddef01db"
Oct 30 20:38:23.945262 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:23.945232    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:38:23.945352 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:23.945302    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:38:23.945352 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:23.945333    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:38:24.367181 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:24.367141    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:24.962177 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:24.962142    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6" event={"ID":"99069007-e0f8-4092-8fd6-f27b0a119720","Type":"ContainerStarted","Data":"6cdaafbc6f4f2066fb222068e8e940e1b0214f283427088b744a0bac7802766e"}
Oct 30 20:38:25.316335 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:25.316299    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:25.751097 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:25.751008    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:38:25.751436 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:25.751354    5071 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 30 20:38:25.751590 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:25.751567    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6" podUID="99069007-e0f8-4092-8fd6-f27b0a119720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 30 20:38:26.390134 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:26.390098    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:26.753354 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:26.753256    5071 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 30 20:38:26.753354 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:26.753321    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6" podUID="99069007-e0f8-4092-8fd6-f27b0a119720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 30 20:38:27.024722 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:27.024635    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:27.673658 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:27.673627    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:27.748846 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:27.748200    5071 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 30 20:38:27.748846 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:27.748260    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6" podUID="99069007-e0f8-4092-8fd6-f27b0a119720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 30 20:38:27.852901 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:27.852864    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:28.751262 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:28.751222    5071 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 30 20:38:28.751914 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:28.751879    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6" podUID="99069007-e0f8-4092-8fd6-f27b0a119720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 30 20:38:29.750850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:29.750224    5071 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 30 20:38:29.750850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:29.750290    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6" podUID="99069007-e0f8-4092-8fd6-f27b0a119720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 30 20:38:30.440711 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:30.440677    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:38:30.751789 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:30.751708    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:38:30.996671 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:30.996647    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:38:31.868301 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:31.868265    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:32.089184 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:32.089151    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:37.129139 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:37.129095    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:37.375067 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:37.375031    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:38:37.582227 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:37.582192    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:38.244221 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:38.244188    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:38.539208 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:38.539169    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:42.321305 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:42.321259    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:43.703738 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:43.703705    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:46.727157 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:38:46.727117    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:38:46.934070 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:46.934034    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:47.607965 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:47.607925    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:47.789397 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:47.789364    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:52.239423 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:52.239390    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:38:52.708553 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:52.708514    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:52.884522 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:52.884477    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:53.590606 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:53.590575    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:54.239957 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:54.239916    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:54.419733 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:54.419698    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:38:58.282607 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:38:58.282572    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:00.160966 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:00.160927    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:03.301099 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:03.301060    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:03.465657 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:03.465617    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:05.465190 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:05.465154    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:39:08.280569 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:08.280487    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:13.371966 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:13.371926    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:16.727474 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:39:16.727434    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:39:18.490320 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:18.490279    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:18.803492 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:18.803455    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:23.487207 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:23.487169    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:23.698175 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:23.698144    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:24.230791 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:24.230757    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:24.757658 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:24.757620    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:25.677830 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:25.677798    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:28.601283 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:28.601251    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:29.303745 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:29.303714    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:39:33.031949 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:33.031914    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"]
Oct 30 20:39:33.036084 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:33.036064    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:39:33.110602 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:33.107251    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-dshm\") pod \"simulator-w0-b32941-kserve-894b678fb-5gzn4\" (UID: \"7ff8df1b-99b0-4b2d-979b-89946b9a16f6\") " pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:39:33.110602 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:33.107308    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-model-cache\") pod \"simulator-w0-b32941-kserve-894b678fb-5gzn4\" (UID: \"7ff8df1b-99b0-4b2d-979b-89946b9a16f6\") " pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:39:33.110602 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:33.107419    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-tmp-dir\") pod \"simulator-w0-b32941-kserve-894b678fb-5gzn4\" (UID: \"7ff8df1b-99b0-4b2d-979b-89946b9a16f6\") " pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:39:33.110602 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:33.107450    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-tls-certs\") pod \"simulator-w0-b32941-kserve-894b678fb-5gzn4\" (UID: \"7ff8df1b-99b0-4b2d-979b-89946b9a16f6\") " pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:39:33.110602 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:33.107504    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-home\") pod \"simulator-w0-b32941-kserve-894b678fb-5gzn4\" (UID: \"7ff8df1b-99b0-4b2d-979b-89946b9a16f6\") " pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:39:33.110602 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:33.107553    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcgdm\" (UniqueName: \"kubernetes.io/projected/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-kube-api-access-lcgdm\") pod \"simulator-w0-b32941-kserve-894b678fb-5gzn4\" (UID: \"7ff8df1b-99b0-4b2d-979b-89946b9a16f6\") " pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:39:33.141287 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:33.141253    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b32941\"/\"kube-root-ca.crt\""
Oct 30 20:39:33.141462 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:33.141289    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b32941\"/\"simulator-w0-b32941-kserve-self-signed-certs\""
Oct 30 20:39:33.141462 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:33.141289    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b32941\"/\"default-dockercfg-gqqws\""
Oct 30 20:39:33.141462 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:33.141253    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b32941\"/\"openshift-service-ca.crt\""
Oct 30 20:39:33.146203 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:33.146176    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"]
Oct 30 20:39:33.160465 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:33.160440    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:33.208758 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:33.208728    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-dshm\") pod \"simulator-w0-b32941-kserve-894b678fb-5gzn4\" (UID: \"7ff8df1b-99b0-4b2d-979b-89946b9a16f6\") " pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:39:33.208758 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:33.208757    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-model-cache\") pod \"simulator-w0-b32941-kserve-894b678fb-5gzn4\" (UID: \"7ff8df1b-99b0-4b2d-979b-89946b9a16f6\") " pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:39:33.209037 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:33.208885    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-tmp-dir\") pod \"simulator-w0-b32941-kserve-894b678fb-5gzn4\" (UID: \"7ff8df1b-99b0-4b2d-979b-89946b9a16f6\") " pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:39:33.209037 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:33.208912    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-tls-certs\") pod \"simulator-w0-b32941-kserve-894b678fb-5gzn4\" (UID: \"7ff8df1b-99b0-4b2d-979b-89946b9a16f6\") " pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:39:33.209037 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:33.208954    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-home\") pod \"simulator-w0-b32941-kserve-894b678fb-5gzn4\" (UID: \"7ff8df1b-99b0-4b2d-979b-89946b9a16f6\") " pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:39:33.209037 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:33.208991    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcgdm\" (UniqueName: \"kubernetes.io/projected/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-kube-api-access-lcgdm\") pod \"simulator-w0-b32941-kserve-894b678fb-5gzn4\" (UID: \"7ff8df1b-99b0-4b2d-979b-89946b9a16f6\") " pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:39:33.209260 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:33.209197    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-model-cache\") pod \"simulator-w0-b32941-kserve-894b678fb-5gzn4\" (UID: \"7ff8df1b-99b0-4b2d-979b-89946b9a16f6\") " pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:39:33.209340 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:33.209313    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-tmp-dir\") pod \"simulator-w0-b32941-kserve-894b678fb-5gzn4\" (UID: \"7ff8df1b-99b0-4b2d-979b-89946b9a16f6\") " pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:39:33.209476 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:33.209453    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-home\") pod \"simulator-w0-b32941-kserve-894b678fb-5gzn4\" (UID: \"7ff8df1b-99b0-4b2d-979b-89946b9a16f6\") " pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:39:33.211417 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:33.211392    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-dshm\") pod \"simulator-w0-b32941-kserve-894b678fb-5gzn4\" (UID: \"7ff8df1b-99b0-4b2d-979b-89946b9a16f6\") " pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:39:33.211660 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:33.211641    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-tls-certs\") pod \"simulator-w0-b32941-kserve-894b678fb-5gzn4\" (UID: \"7ff8df1b-99b0-4b2d-979b-89946b9a16f6\") " pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:39:33.269457 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:33.269426    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcgdm\" (UniqueName: \"kubernetes.io/projected/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-kube-api-access-lcgdm\") pod \"simulator-w0-b32941-kserve-894b678fb-5gzn4\" (UID: \"7ff8df1b-99b0-4b2d-979b-89946b9a16f6\") " pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:39:33.375345 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:33.375305    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:39:33.606678 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:33.606641    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:33.755745 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:33.755720    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"]
Oct 30 20:39:33.757322 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:39:33.757299    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ff8df1b_99b0_4b2d_979b_89946b9a16f6.slice/crio-d3a369b928e69f7eb0840bc9725124418ab1f5f81f6af1b28a81146de4aebd78 WatchSource:0}: Error finding container d3a369b928e69f7eb0840bc9725124418ab1f5f81f6af1b28a81146de4aebd78: Status 404 returned error can't find the container with id d3a369b928e69f7eb0840bc9725124418ab1f5f81f6af1b28a81146de4aebd78
Oct 30 20:39:34.376438 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:34.376398    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" event={"ID":"7ff8df1b-99b0-4b2d-979b-89946b9a16f6","Type":"ContainerStarted","Data":"2d151d574b82c2958e5dd9c5dde138f114fb177e3b62e42663244b57d101dd7a"}
Oct 30 20:39:34.376851 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:34.376451    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" event={"ID":"7ff8df1b-99b0-4b2d-979b-89946b9a16f6","Type":"ContainerStarted","Data":"d3a369b928e69f7eb0840bc9725124418ab1f5f81f6af1b28a81146de4aebd78"}
Oct 30 20:39:34.494593 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:34.494534    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podStartSLOduration=2.494518559 podStartE2EDuration="2.494518559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:34.492895149 +0000 UTC m=+786.409916687" watchObservedRunningTime="2026-10-30 20:39:34.494518559 +0000 UTC m=+786.411540094"
Oct 30 20:39:35.375783 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:35.375748    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:39:36.571304 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:36.571262    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:36.803842 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:36.803797    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:38.705143 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:38.705103    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:43.375893 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:43.375855    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:39:43.389119 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:43.389087    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:39:43.440676 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:43.440650    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:39:43.773407 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:43.773323    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:46.562034 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:46.561979    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:46.727700 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:39:46.727663    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:39:48.871158 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:48.871120    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:52.895585 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:52.895550    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:39:54.024148 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:54.024083    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:58.933768 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:58.933720    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:39:59.186574 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:39:59.185540    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:40:03.981687 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:03.981647    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:40:08.997631 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:08.997592    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:40:13.984302 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:13.984257    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:40:15.395952 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:15.395915    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:40:16.727563 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:40:16.727529    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:40:17.763095 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:17.763065    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"]
Oct 30 20:40:17.767329 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:17.767307    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:40:17.786008 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:17.785955    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7919394d-614c-4f2d-b04a-f44f90d56e96-tmp-dir\") pod \"premium-w4-395022-kserve-5d7d5658f7-kgjvr\" (UID: \"7919394d-614c-4f2d-b04a-f44f90d56e96\") " pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:40:17.786424 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:17.786399    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7919394d-614c-4f2d-b04a-f44f90d56e96-home\") pod \"premium-w4-395022-kserve-5d7d5658f7-kgjvr\" (UID: \"7919394d-614c-4f2d-b04a-f44f90d56e96\") " pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:40:17.786535 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:17.786433    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7919394d-614c-4f2d-b04a-f44f90d56e96-dshm\") pod \"premium-w4-395022-kserve-5d7d5658f7-kgjvr\" (UID: \"7919394d-614c-4f2d-b04a-f44f90d56e96\") " pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:40:17.786535 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:17.786477    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx4ch\" (UniqueName: \"kubernetes.io/projected/7919394d-614c-4f2d-b04a-f44f90d56e96-kube-api-access-wx4ch\") pod \"premium-w4-395022-kserve-5d7d5658f7-kgjvr\" (UID: \"7919394d-614c-4f2d-b04a-f44f90d56e96\") " pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:40:17.786706 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:17.786557    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7919394d-614c-4f2d-b04a-f44f90d56e96-tls-certs\") pod \"premium-w4-395022-kserve-5d7d5658f7-kgjvr\" (UID: \"7919394d-614c-4f2d-b04a-f44f90d56e96\") " pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:40:17.786706 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:17.786594    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7919394d-614c-4f2d-b04a-f44f90d56e96-model-cache\") pod \"premium-w4-395022-kserve-5d7d5658f7-kgjvr\" (UID: \"7919394d-614c-4f2d-b04a-f44f90d56e96\") " pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:40:17.871500 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:17.871453    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-395022\"/\"openshift-service-ca.crt\""
Oct 30 20:40:17.871698 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:17.871513    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-395022\"/\"premium-w4-395022-kserve-self-signed-certs\""
Oct 30 20:40:17.871879 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:17.871855    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-395022\"/\"kube-root-ca.crt\""
Oct 30 20:40:17.872003 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:17.871969    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-395022\"/\"default-dockercfg-gqktn\""
Oct 30 20:40:17.874156 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:17.874135    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"]
Oct 30 20:40:17.881283 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:17.881261    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:40:17.887140 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:17.887121    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx4ch\" (UniqueName: \"kubernetes.io/projected/7919394d-614c-4f2d-b04a-f44f90d56e96-kube-api-access-wx4ch\") pod \"premium-w4-395022-kserve-5d7d5658f7-kgjvr\" (UID: \"7919394d-614c-4f2d-b04a-f44f90d56e96\") " pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:40:17.887246 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:17.887150    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7919394d-614c-4f2d-b04a-f44f90d56e96-tls-certs\") pod \"premium-w4-395022-kserve-5d7d5658f7-kgjvr\" (UID: \"7919394d-614c-4f2d-b04a-f44f90d56e96\") " pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:40:17.887246 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:17.887172    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7919394d-614c-4f2d-b04a-f44f90d56e96-model-cache\") pod \"premium-w4-395022-kserve-5d7d5658f7-kgjvr\" (UID: \"7919394d-614c-4f2d-b04a-f44f90d56e96\") " pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:40:17.887246 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:17.887218    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7919394d-614c-4f2d-b04a-f44f90d56e96-tmp-dir\") pod \"premium-w4-395022-kserve-5d7d5658f7-kgjvr\" (UID: \"7919394d-614c-4f2d-b04a-f44f90d56e96\") " pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:40:17.887368 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:17.887304    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7919394d-614c-4f2d-b04a-f44f90d56e96-home\") pod \"premium-w4-395022-kserve-5d7d5658f7-kgjvr\" (UID: \"7919394d-614c-4f2d-b04a-f44f90d56e96\") " pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:40:17.887368 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:17.887326    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7919394d-614c-4f2d-b04a-f44f90d56e96-dshm\") pod \"premium-w4-395022-kserve-5d7d5658f7-kgjvr\" (UID: \"7919394d-614c-4f2d-b04a-f44f90d56e96\") " pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:40:17.887676 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:17.887656    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7919394d-614c-4f2d-b04a-f44f90d56e96-tmp-dir\") pod \"premium-w4-395022-kserve-5d7d5658f7-kgjvr\" (UID: \"7919394d-614c-4f2d-b04a-f44f90d56e96\") " pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:40:17.887774 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:17.887716    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7919394d-614c-4f2d-b04a-f44f90d56e96-home\") pod \"premium-w4-395022-kserve-5d7d5658f7-kgjvr\" (UID: \"7919394d-614c-4f2d-b04a-f44f90d56e96\") " pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:40:17.887774 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:17.887751    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7919394d-614c-4f2d-b04a-f44f90d56e96-model-cache\") pod \"premium-w4-395022-kserve-5d7d5658f7-kgjvr\" (UID: \"7919394d-614c-4f2d-b04a-f44f90d56e96\") " pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:40:17.890205 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:17.890186    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7919394d-614c-4f2d-b04a-f44f90d56e96-dshm\") pod \"premium-w4-395022-kserve-5d7d5658f7-kgjvr\" (UID: \"7919394d-614c-4f2d-b04a-f44f90d56e96\") " pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:40:17.890368 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:17.890353    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7919394d-614c-4f2d-b04a-f44f90d56e96-tls-certs\") pod \"premium-w4-395022-kserve-5d7d5658f7-kgjvr\" (UID: \"7919394d-614c-4f2d-b04a-f44f90d56e96\") " pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:40:17.970272 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:17.970219    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx4ch\" (UniqueName: \"kubernetes.io/projected/7919394d-614c-4f2d-b04a-f44f90d56e96-kube-api-access-wx4ch\") pod \"premium-w4-395022-kserve-5d7d5658f7-kgjvr\" (UID: \"7919394d-614c-4f2d-b04a-f44f90d56e96\") " pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:40:18.079653 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:18.079623    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:40:18.487318 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:18.487294    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"]
Oct 30 20:40:18.488931 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:40:18.488905    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7919394d_614c_4f2d_b04a_f44f90d56e96.slice/crio-e454e2d83d28b90d243709bf7133fb4437f34457e63a4d5bf75e4d65470dbc6e WatchSource:0}: Error finding container e454e2d83d28b90d243709bf7133fb4437f34457e63a4d5bf75e4d65470dbc6e: Status 404 returned error can't find the container with id e454e2d83d28b90d243709bf7133fb4437f34457e63a4d5bf75e4d65470dbc6e
Oct 30 20:40:18.644776 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:18.644742    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" event={"ID":"7919394d-614c-4f2d-b04a-f44f90d56e96","Type":"ContainerStarted","Data":"4974a19737d331919de23e6f3ff07bd90ab7a0b1ff4b6bb64986d5edd1ce354f"}
Oct 30 20:40:18.644776 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:18.644780    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" event={"ID":"7919394d-614c-4f2d-b04a-f44f90d56e96","Type":"ContainerStarted","Data":"e454e2d83d28b90d243709bf7133fb4437f34457e63a4d5bf75e4d65470dbc6e"}
Oct 30 20:40:18.761324 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:18.760782    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podStartSLOduration=1.760766554 podStartE2EDuration="1.760766554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:40:18.760552421 +0000 UTC m=+830.677573955" watchObservedRunningTime="2026-10-30 20:40:18.760766554 +0000 UTC m=+830.677788089"
Oct 30 20:40:18.887892 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:18.887855    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:40:19.080604 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:19.080569    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:40:20.279337 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:20.279306    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:40:23.792718 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:23.792675    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:40:23.991411 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:23.991376    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:40:25.070029 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:25.069989    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:40:27.061126 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:27.061085    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:40:28.080843 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:28.080791    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:40:28.094114 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:28.094086    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:40:28.689178 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:28.689139    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:40:28.738290 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:28.738243    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:40:33.577614 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:33.577574    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:40:36.446030 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:36.445990    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"]
Oct 30 20:40:36.451255 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:36.451232    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:40:36.510592 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:36.510557    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"]
Oct 30 20:40:36.555119 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:36.555083    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-386743\"/\"kube-root-ca.crt\""
Oct 30 20:40:36.555304 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:36.555285    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-386743\"/\"openshift-service-ca.crt\""
Oct 30 20:40:36.555400 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:36.555373    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-386743\"/\"premium-w5-386743-kserve-self-signed-certs\""
Oct 30 20:40:36.555448 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:36.555395    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-386743\"/\"default-dockercfg-7mljt\""
Oct 30 20:40:36.563596 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:36.563569    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:40:36.584251 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:36.584221    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvc6m\" (UniqueName: \"kubernetes.io/projected/404848bf-abd8-421c-9419-57cb32fe6ef4-kube-api-access-rvc6m\") pod \"premium-w5-386743-kserve-cdc94b7fd-v9c94\" (UID: \"404848bf-abd8-421c-9419-57cb32fe6ef4\") " pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:40:36.584364 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:36.584263    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/404848bf-abd8-421c-9419-57cb32fe6ef4-home\") pod \"premium-w5-386743-kserve-cdc94b7fd-v9c94\" (UID: \"404848bf-abd8-421c-9419-57cb32fe6ef4\") " pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:40:36.584364 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:36.584292    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/404848bf-abd8-421c-9419-57cb32fe6ef4-model-cache\") pod \"premium-w5-386743-kserve-cdc94b7fd-v9c94\" (UID: \"404848bf-abd8-421c-9419-57cb32fe6ef4\") " pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:40:36.584473 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:36.584370    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/404848bf-abd8-421c-9419-57cb32fe6ef4-tmp-dir\") pod \"premium-w5-386743-kserve-cdc94b7fd-v9c94\" (UID: \"404848bf-abd8-421c-9419-57cb32fe6ef4\") " pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:40:36.584473 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:36.584446    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/404848bf-abd8-421c-9419-57cb32fe6ef4-dshm\") pod \"premium-w5-386743-kserve-cdc94b7fd-v9c94\" (UID: \"404848bf-abd8-421c-9419-57cb32fe6ef4\") " pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:40:36.584560 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:36.584541    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/404848bf-abd8-421c-9419-57cb32fe6ef4-tls-certs\") pod \"premium-w5-386743-kserve-cdc94b7fd-v9c94\" (UID: \"404848bf-abd8-421c-9419-57cb32fe6ef4\") " pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:40:36.686002 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:36.685963    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/404848bf-abd8-421c-9419-57cb32fe6ef4-tls-certs\") pod \"premium-w5-386743-kserve-cdc94b7fd-v9c94\" (UID: \"404848bf-abd8-421c-9419-57cb32fe6ef4\") " pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:40:36.686256 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:36.686074    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvc6m\" (UniqueName: \"kubernetes.io/projected/404848bf-abd8-421c-9419-57cb32fe6ef4-kube-api-access-rvc6m\") pod \"premium-w5-386743-kserve-cdc94b7fd-v9c94\" (UID: \"404848bf-abd8-421c-9419-57cb32fe6ef4\") " pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:40:36.686256 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:36.686107    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/404848bf-abd8-421c-9419-57cb32fe6ef4-home\") pod \"premium-w5-386743-kserve-cdc94b7fd-v9c94\" (UID: \"404848bf-abd8-421c-9419-57cb32fe6ef4\") " pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:40:36.686256 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:36.686140    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/404848bf-abd8-421c-9419-57cb32fe6ef4-model-cache\") pod \"premium-w5-386743-kserve-cdc94b7fd-v9c94\" (UID: \"404848bf-abd8-421c-9419-57cb32fe6ef4\") " pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:40:36.686256 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:36.686163    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/404848bf-abd8-421c-9419-57cb32fe6ef4-tmp-dir\") pod \"premium-w5-386743-kserve-cdc94b7fd-v9c94\" (UID: \"404848bf-abd8-421c-9419-57cb32fe6ef4\") " pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:40:36.686256 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:36.686195    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/404848bf-abd8-421c-9419-57cb32fe6ef4-dshm\") pod \"premium-w5-386743-kserve-cdc94b7fd-v9c94\" (UID: \"404848bf-abd8-421c-9419-57cb32fe6ef4\") " pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:40:36.686543 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:36.686517    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/404848bf-abd8-421c-9419-57cb32fe6ef4-home\") pod \"premium-w5-386743-kserve-cdc94b7fd-v9c94\" (UID: \"404848bf-abd8-421c-9419-57cb32fe6ef4\") " pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:40:36.686690 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:36.686648    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/404848bf-abd8-421c-9419-57cb32fe6ef4-model-cache\") pod \"premium-w5-386743-kserve-cdc94b7fd-v9c94\" (UID: \"404848bf-abd8-421c-9419-57cb32fe6ef4\") " pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:40:36.686690 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:36.686659    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/404848bf-abd8-421c-9419-57cb32fe6ef4-tmp-dir\") pod \"premium-w5-386743-kserve-cdc94b7fd-v9c94\" (UID: \"404848bf-abd8-421c-9419-57cb32fe6ef4\") " pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:40:36.688647 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:36.688624    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/404848bf-abd8-421c-9419-57cb32fe6ef4-tls-certs\") pod \"premium-w5-386743-kserve-cdc94b7fd-v9c94\" (UID: \"404848bf-abd8-421c-9419-57cb32fe6ef4\") " pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:40:36.688741 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:36.688685    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/404848bf-abd8-421c-9419-57cb32fe6ef4-dshm\") pod \"premium-w5-386743-kserve-cdc94b7fd-v9c94\" (UID: \"404848bf-abd8-421c-9419-57cb32fe6ef4\") " pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:40:36.744255 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:36.744184    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvc6m\" (UniqueName: \"kubernetes.io/projected/404848bf-abd8-421c-9419-57cb32fe6ef4-kube-api-access-rvc6m\") pod \"premium-w5-386743-kserve-cdc94b7fd-v9c94\" (UID: \"404848bf-abd8-421c-9419-57cb32fe6ef4\") " pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:40:36.797353 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:36.797313    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:40:37.196792 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:37.196768    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"]
Oct 30 20:40:37.198199 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:40:37.198172    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod404848bf_abd8_421c_9419_57cb32fe6ef4.slice/crio-379dfae7f35ec30bca15fe65272bb74b7372477068294d8a05afddf207e4d94c WatchSource:0}: Error finding container 379dfae7f35ec30bca15fe65272bb74b7372477068294d8a05afddf207e4d94c: Status 404 returned error can't find the container with id 379dfae7f35ec30bca15fe65272bb74b7372477068294d8a05afddf207e4d94c
Oct 30 20:40:37.764302 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:37.764262    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" event={"ID":"404848bf-abd8-421c-9419-57cb32fe6ef4","Type":"ContainerStarted","Data":"e2f407d557e8c4abddacbcfcf5d08d1ef55215a27b213f82c80f7836333a9a91"}
Oct 30 20:40:37.764302 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:37.764305    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" event={"ID":"404848bf-abd8-421c-9419-57cb32fe6ef4","Type":"ContainerStarted","Data":"379dfae7f35ec30bca15fe65272bb74b7372477068294d8a05afddf207e4d94c"}
Oct 30 20:40:37.797416 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:37.797389    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:40:37.881887 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:37.881808    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podStartSLOduration=1.881791665 podStartE2EDuration="1.881791665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:40:37.880094767 +0000 UTC m=+849.797116301" watchObservedRunningTime="2026-10-30 20:40:37.881791665 +0000 UTC m=+849.798813201"
Oct 30 20:40:38.706420 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:38.706382    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:40:42.201203 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:42.201165    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:40:43.600800 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:43.600766    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:40:45.008580 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:45.008541    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-683f3132-5bq2k"]
Oct 30 20:40:45.013081 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:45.013056    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-683f3132-5bq2k"
Oct 30 20:40:45.070546 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:45.070512    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-683f3132-5bq2k"]
Oct 30 20:40:45.101225 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:45.101194    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-683f3132-5bq2k\" (UID: \"9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683f3132-5bq2k"
Oct 30 20:40:45.101385 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:45.101248    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b25jh\" (UniqueName: \"kubernetes.io/projected/9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55-kube-api-access-b25jh\") pod \"maas-api-revoke-sub-683f3132-5bq2k\" (UID: \"9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683f3132-5bq2k"
Oct 30 20:40:45.101428 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:45.101378    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-683f3132-5bq2k\" (UID: \"9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683f3132-5bq2k"
Oct 30 20:40:45.117075 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:45.117046    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:40:45.117075 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:45.117058    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 20:40:45.202664 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:45.202629    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-683f3132-5bq2k\" (UID: \"9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683f3132-5bq2k"
Oct 30 20:40:45.202912 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:45.202688    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b25jh\" (UniqueName: \"kubernetes.io/projected/9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55-kube-api-access-b25jh\") pod \"maas-api-revoke-sub-683f3132-5bq2k\" (UID: \"9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683f3132-5bq2k"
Oct 30 20:40:45.202912 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:45.202724    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-683f3132-5bq2k\" (UID: \"9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683f3132-5bq2k"
Oct 30 20:40:45.203542 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:45.203514    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-683f3132-5bq2k\" (UID: \"9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683f3132-5bq2k"
Oct 30 20:40:45.205337 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:45.205311    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-683f3132-5bq2k\" (UID: \"9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683f3132-5bq2k"
Oct 30 20:40:45.264789 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:45.264703    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b25jh\" (UniqueName: \"kubernetes.io/projected/9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55-kube-api-access-b25jh\") pod \"maas-api-revoke-sub-683f3132-5bq2k\" (UID: \"9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683f3132-5bq2k"
Oct 30 20:40:45.326851 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:45.326797    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-683f3132-5bq2k"
Oct 30 20:40:45.700863 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:45.700810    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-683f3132-5bq2k"]
Oct 30 20:40:45.702103 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:40:45.702073    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e7aa51a_7ae4_4ba1_a4c7_1079c70f3b55.slice/crio-03dbbbf215179168bdd792b7843cfa9aed720393ca82cbdf0ece918be9d519b4 WatchSource:0}: Error finding container 03dbbbf215179168bdd792b7843cfa9aed720393ca82cbdf0ece918be9d519b4: Status 404 returned error can't find the container with id 03dbbbf215179168bdd792b7843cfa9aed720393ca82cbdf0ece918be9d519b4
Oct 30 20:40:45.816018 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:45.815982    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-683f3132-5bq2k" event={"ID":"9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55","Type":"ContainerStarted","Data":"03dbbbf215179168bdd792b7843cfa9aed720393ca82cbdf0ece918be9d519b4"}
Oct 30 20:40:46.729128 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:40:46.729084    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:40:46.798425 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:46.798395    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:40:46.811366 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:46.811340    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:40:46.832776 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:46.832752    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:40:48.580648 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:48.580613    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:40:53.453044 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:53.453006    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:40:53.673619 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:53.673585    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:40:58.493783 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:40:58.493749    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:03.639321 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:03.639234    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:03.935251 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:03.935154    5071 generic.go:356] "Generic (PLEG): container finished" podID="9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55" containerID="73c5064ecab9e10f7e206920e193e48a9f7ae3561df213cc555b6bb7894a3d45" exitCode=0
Oct 30 20:41:03.935399 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:03.935243    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-683f3132-5bq2k" event={"ID":"9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55","Type":"ContainerDied","Data":"73c5064ecab9e10f7e206920e193e48a9f7ae3561df213cc555b6bb7894a3d45"}
Oct 30 20:41:05.097102 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:05.097074    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-683f3132-5bq2k"
Oct 30 20:41:05.225235 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:05.225156    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55-maas-api-openshift-ca\") pod \"9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55\" (UID: \"9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55\") "
Oct 30 20:41:05.225404 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:05.225250    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55-maas-api-xks-ca\") pod \"9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55\" (UID: \"9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55\") "
Oct 30 20:41:05.225484 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:05.225437    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55-kube-api-access-b25jh\" (UniqueName: \"kubernetes.io/projected/9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55-kube-api-access-b25jh\") pod \"9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55\" (UID: \"9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55\") "
Oct 30 20:41:05.225671 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:05.225635    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55-maas-api-openshift-ca" pod "9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55" (UID: "9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:41:05.225924 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:05.225901    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:05.227892 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:05.227864    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55-maas-api-xks-ca" pod "9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55" (UID: "9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:41:05.228052 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:05.228029    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55-kube-api-access-b25jh" pod "9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55" (UID: "9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55"). InnerVolumeSpecName "kube-api-access-b25jh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:41:05.326663 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:05.326628    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b25jh\" (UniqueName: \"kubernetes.io/projected/9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55-kube-api-access-b25jh\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:05.326663 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:05.326657    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:05.948770 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:05.948738    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-683f3132-5bq2k"
Oct 30 20:41:05.948770 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:05.948751    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-683f3132-5bq2k" event={"ID":"9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55","Type":"ContainerDied","Data":"03dbbbf215179168bdd792b7843cfa9aed720393ca82cbdf0ece918be9d519b4"}
Oct 30 20:41:05.949034 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:05.948797    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03dbbbf215179168bdd792b7843cfa9aed720393ca82cbdf0ece918be9d519b4"
Oct 30 20:41:08.631651 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:08.631616    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:10.006139 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:10.006099    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:13.308089 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:13.308046    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"]
Oct 30 20:41:13.308909 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:13.308880    5071 cpu_manager.go:414] "containerMap: removing container" podUID="9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55" containerName="revoke-keys"
Oct 30 20:41:13.308909 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:13.308913    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55" containerName="revoke-keys"
Oct 30 20:41:13.309056 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:13.309043    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55" containerName="revoke-keys"
Oct 30 20:41:13.312111 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:13.312094    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"
Oct 30 20:41:13.317763 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:13.317735    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-credential-socket\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l\" (UID: \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"
Oct 30 20:41:13.317922 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:13.317771    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-istio-envoy\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l\" (UID: \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"
Oct 30 20:41:13.317922 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:13.317796    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-istio-token\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l\" (UID: \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"
Oct 30 20:41:13.318037 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:13.317953    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wr7j\" (UniqueName: \"kubernetes.io/projected/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-kube-api-access-5wr7j\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l\" (UID: \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"
Oct 30 20:41:13.318037 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:13.317995    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-istio-podinfo\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l\" (UID: \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"
Oct 30 20:41:13.318037 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:13.318014    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-workload-certs\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l\" (UID: \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"
Oct 30 20:41:13.318191 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:13.318102    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-istiod-ca-cert\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l\" (UID: \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"
Oct 30 20:41:13.318191 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:13.318131    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-workload-socket\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l\" (UID: \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"
Oct 30 20:41:13.318191 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:13.318169    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-istio-data\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l\" (UID: \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"
Oct 30 20:41:13.366004 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:13.365974    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-386743-openshift-default-dockercfg-zfbp9\""
Oct 30 20:41:13.419184 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:13.419147    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wr7j\" (UniqueName: \"kubernetes.io/projected/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-kube-api-access-5wr7j\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l\" (UID: \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"
Oct 30 20:41:13.419391 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:13.419217    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-istio-podinfo\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l\" (UID: \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"
Oct 30 20:41:13.419391 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:13.419243    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-workload-certs\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l\" (UID: \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"
Oct 30 20:41:13.419391 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:13.419293    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-istiod-ca-cert\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l\" (UID: \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"
Oct 30 20:41:13.419391 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:13.419318    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-workload-socket\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l\" (UID: \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"
Oct 30 20:41:13.419391 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:13.419365    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-istio-data\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l\" (UID: \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"
Oct 30 20:41:13.419645 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:13.419402    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-credential-socket\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l\" (UID: \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"
Oct 30 20:41:13.419645 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:13.419429    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-istio-envoy\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l\" (UID: \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"
Oct 30 20:41:13.419645 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:13.419461    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-istio-token\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l\" (UID: \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"
Oct 30 20:41:13.419645 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:13.419536    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"]
Oct 30 20:41:13.420420 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:13.420393    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-istio-data\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l\" (UID: \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"
Oct 30 20:41:13.420733 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:13.420707    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-workload-certs\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l\" (UID: \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"
Oct 30 20:41:13.421123 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:13.420972    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-workload-socket\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l\" (UID: \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"
Oct 30 20:41:13.421223 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:13.421200    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-credential-socket\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l\" (UID: \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"
Oct 30 20:41:13.421807 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:13.421785    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-istiod-ca-cert\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l\" (UID: \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"
Oct 30 20:41:13.423324 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:13.423298    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-istio-podinfo\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l\" (UID: \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"
Oct 30 20:41:13.423909 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:13.423886    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-istio-envoy\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l\" (UID: \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"
Oct 30 20:41:13.479860 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:13.479812    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-istio-token\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l\" (UID: \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"
Oct 30 20:41:13.480020 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:13.479991    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wr7j\" (UniqueName: \"kubernetes.io/projected/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-kube-api-access-5wr7j\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l\" (UID: \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"
Oct 30 20:41:13.626201 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:13.626107    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"
Oct 30 20:41:13.776333 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:13.776296    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:14.056916 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:14.056890    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"]
Oct 30 20:41:14.057978 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:41:14.057938    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10329fd9_4ea0_41ac_a93b_6a0237b5f7c1.slice/crio-0285fee41de1147c18f87a0f78e07ba001483102763ad44961084303fe2388f0 WatchSource:0}: Error finding container 0285fee41de1147c18f87a0f78e07ba001483102763ad44961084303fe2388f0: Status 404 returned error can't find the container with id 0285fee41de1147c18f87a0f78e07ba001483102763ad44961084303fe2388f0
Oct 30 20:41:14.060229 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:14.060195    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:41:14.060318 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:14.060259    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:41:14.060318 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:14.060286    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:41:15.007478 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:15.007441    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l" event={"ID":"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1","Type":"ContainerStarted","Data":"554e99148efcd61cf3d5a180bdaf3ee851a735139dce85de7546fcf330fda033"}
Oct 30 20:41:15.007931 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:15.007498    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l" event={"ID":"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1","Type":"ContainerStarted","Data":"0285fee41de1147c18f87a0f78e07ba001483102763ad44961084303fe2388f0"}
Oct 30 20:41:15.226052 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:15.225999    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l" podStartSLOduration=2.225985586 podStartE2EDuration="2.225985586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:15.174774693 +0000 UTC m=+887.091796231" watchObservedRunningTime="2026-10-30 20:41:15.225985586 +0000 UTC m=+887.143007122"
Oct 30 20:41:15.626888 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:15.626857    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"
Oct 30 20:41:15.634847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:15.629139    5071 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 20:41:15.634847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:15.629202    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l" podUID="10329fd9-4ea0-41ac-a93b-6a0237b5f7c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 20:41:16.627129 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:16.627093    5071 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 20:41:16.627538 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:16.627151    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l" podUID="10329fd9-4ea0-41ac-a93b-6a0237b5f7c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 20:41:16.753601 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:41:16.753552    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:41:17.053003 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:17.052960    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adc2708d-dbrhj"]
Oct 30 20:41:17.056996 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:17.056979    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adc2708d-dbrhj"
Oct 30 20:41:17.080338 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:17.080308    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/544da439-8c62-47c7-bdb5-256b8b55a890-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adc2708d-dbrhj\" (UID: \"544da439-8c62-47c7-bdb5-256b8b55a890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adc2708d-dbrhj"
Oct 30 20:41:17.080462 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:17.080376    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96k5w\" (UniqueName: \"kubernetes.io/projected/544da439-8c62-47c7-bdb5-256b8b55a890-kube-api-access-96k5w\") pod \"maas-api-revoke-sub-adc2708d-dbrhj\" (UID: \"544da439-8c62-47c7-bdb5-256b8b55a890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adc2708d-dbrhj"
Oct 30 20:41:17.080462 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:17.080417    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/544da439-8c62-47c7-bdb5-256b8b55a890-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adc2708d-dbrhj\" (UID: \"544da439-8c62-47c7-bdb5-256b8b55a890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adc2708d-dbrhj"
Oct 30 20:41:17.096645 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:17.096620    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adc2708d-dbrhj"]
Oct 30 20:41:17.147845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:17.143065    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:41:17.147845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:17.143210    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 20:41:17.181904 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:17.181868    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96k5w\" (UniqueName: \"kubernetes.io/projected/544da439-8c62-47c7-bdb5-256b8b55a890-kube-api-access-96k5w\") pod \"maas-api-revoke-sub-adc2708d-dbrhj\" (UID: \"544da439-8c62-47c7-bdb5-256b8b55a890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adc2708d-dbrhj"
Oct 30 20:41:17.182103 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:17.181938    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/544da439-8c62-47c7-bdb5-256b8b55a890-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adc2708d-dbrhj\" (UID: \"544da439-8c62-47c7-bdb5-256b8b55a890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adc2708d-dbrhj"
Oct 30 20:41:17.182103 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:17.182047    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/544da439-8c62-47c7-bdb5-256b8b55a890-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adc2708d-dbrhj\" (UID: \"544da439-8c62-47c7-bdb5-256b8b55a890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adc2708d-dbrhj"
Oct 30 20:41:17.182606 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:17.182581    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/544da439-8c62-47c7-bdb5-256b8b55a890-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adc2708d-dbrhj\" (UID: \"544da439-8c62-47c7-bdb5-256b8b55a890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adc2708d-dbrhj"
Oct 30 20:41:17.184592 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:17.184567    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/544da439-8c62-47c7-bdb5-256b8b55a890-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adc2708d-dbrhj\" (UID: \"544da439-8c62-47c7-bdb5-256b8b55a890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adc2708d-dbrhj"
Oct 30 20:41:17.257635 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:17.257601    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96k5w\" (UniqueName: \"kubernetes.io/projected/544da439-8c62-47c7-bdb5-256b8b55a890-kube-api-access-96k5w\") pod \"maas-api-revoke-sub-adc2708d-dbrhj\" (UID: \"544da439-8c62-47c7-bdb5-256b8b55a890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adc2708d-dbrhj"
Oct 30 20:41:17.371370 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:17.371285    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adc2708d-dbrhj"
Oct 30 20:41:17.627360 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:17.627280    5071 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 20:41:17.627360 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:17.627342    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l" podUID="10329fd9-4ea0-41ac-a93b-6a0237b5f7c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 20:41:17.785773 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:17.785741    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adc2708d-dbrhj"]
Oct 30 20:41:17.787899 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:41:17.787873    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod544da439_8c62_47c7_bdb5_256b8b55a890.slice/crio-11d0310d34bcef3116ee9625266ae8fb7095bfba0984578532ea66051259aab2 WatchSource:0}: Error finding container 11d0310d34bcef3116ee9625266ae8fb7095bfba0984578532ea66051259aab2: Status 404 returned error can't find the container with id 11d0310d34bcef3116ee9625266ae8fb7095bfba0984578532ea66051259aab2
Oct 30 20:41:18.026774 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:18.026741    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adc2708d-dbrhj" event={"ID":"544da439-8c62-47c7-bdb5-256b8b55a890","Type":"ContainerStarted","Data":"11d0310d34bcef3116ee9625266ae8fb7095bfba0984578532ea66051259aab2"}
Oct 30 20:41:18.627176 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:18.627147    5071 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 20:41:18.627353 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:18.627195    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l" podUID="10329fd9-4ea0-41ac-a93b-6a0237b5f7c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 20:41:18.865304 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:18.865267    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:19.050847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:19.049715    5071 generic.go:356] "Generic (PLEG): container finished" podID="544da439-8c62-47c7-bdb5-256b8b55a890" containerID="fded835e8bcc3b3685900599d605c285be4b29e5845c370cbecb316d6dea0d44" exitCode=0
Oct 30 20:41:19.057848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:19.054997    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adc2708d-dbrhj" event={"ID":"544da439-8c62-47c7-bdb5-256b8b55a890","Type":"ContainerDied","Data":"fded835e8bcc3b3685900599d605c285be4b29e5845c370cbecb316d6dea0d44"}
Oct 30 20:41:19.627266 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:19.627232    5071 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 20:41:19.627450 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:19.627294    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l" podUID="10329fd9-4ea0-41ac-a93b-6a0237b5f7c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 20:41:20.256371 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:20.256343    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adc2708d-dbrhj"
Oct 30 20:41:20.317356 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:20.317325    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/544da439-8c62-47c7-bdb5-256b8b55a890-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/544da439-8c62-47c7-bdb5-256b8b55a890-maas-api-openshift-ca\") pod \"544da439-8c62-47c7-bdb5-256b8b55a890\" (UID: \"544da439-8c62-47c7-bdb5-256b8b55a890\") "
Oct 30 20:41:20.317566 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:20.317431    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/544da439-8c62-47c7-bdb5-256b8b55a890-kube-api-access-96k5w\" (UniqueName: \"kubernetes.io/projected/544da439-8c62-47c7-bdb5-256b8b55a890-kube-api-access-96k5w\") pod \"544da439-8c62-47c7-bdb5-256b8b55a890\" (UID: \"544da439-8c62-47c7-bdb5-256b8b55a890\") "
Oct 30 20:41:20.317566 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:20.317524    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/544da439-8c62-47c7-bdb5-256b8b55a890-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/544da439-8c62-47c7-bdb5-256b8b55a890-maas-api-xks-ca\") pod \"544da439-8c62-47c7-bdb5-256b8b55a890\" (UID: \"544da439-8c62-47c7-bdb5-256b8b55a890\") "
Oct 30 20:41:20.317685 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:20.317658    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/544da439-8c62-47c7-bdb5-256b8b55a890-maas-api-openshift-ca" pod "544da439-8c62-47c7-bdb5-256b8b55a890" (UID: "544da439-8c62-47c7-bdb5-256b8b55a890"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:41:20.317915 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:20.317898    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/544da439-8c62-47c7-bdb5-256b8b55a890-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:20.319660 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:20.319634    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/544da439-8c62-47c7-bdb5-256b8b55a890-maas-api-xks-ca" pod "544da439-8c62-47c7-bdb5-256b8b55a890" (UID: "544da439-8c62-47c7-bdb5-256b8b55a890"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:41:20.319660 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:20.319653    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/544da439-8c62-47c7-bdb5-256b8b55a890-kube-api-access-96k5w" pod "544da439-8c62-47c7-bdb5-256b8b55a890" (UID: "544da439-8c62-47c7-bdb5-256b8b55a890"). InnerVolumeSpecName "kube-api-access-96k5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:41:20.419205 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:20.419170    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/544da439-8c62-47c7-bdb5-256b8b55a890-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:20.419371 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:20.419210    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96k5w\" (UniqueName: \"kubernetes.io/projected/544da439-8c62-47c7-bdb5-256b8b55a890-kube-api-access-96k5w\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:20.810346 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:20.810311    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"
Oct 30 20:41:20.810728 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:20.810699    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"
Oct 30 20:41:20.811640 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:20.811613    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"
Oct 30 20:41:21.085026 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:21.084928    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adc2708d-dbrhj" event={"ID":"544da439-8c62-47c7-bdb5-256b8b55a890","Type":"ContainerDied","Data":"11d0310d34bcef3116ee9625266ae8fb7095bfba0984578532ea66051259aab2"}
Oct 30 20:41:21.085026 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:21.085020    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11d0310d34bcef3116ee9625266ae8fb7095bfba0984578532ea66051259aab2"
Oct 30 20:41:21.085265 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:21.085172    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adc2708d-dbrhj"
Oct 30 20:41:21.902850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:21.902802    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7b2ab73-wfflr"]
Oct 30 20:41:21.903880 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:21.903836    5071 cpu_manager.go:414] "containerMap: removing container" podUID="544da439-8c62-47c7-bdb5-256b8b55a890" containerName="revoke-keys"
Oct 30 20:41:21.903880 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:21.903860    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="544da439-8c62-47c7-bdb5-256b8b55a890" containerName="revoke-keys"
Oct 30 20:41:21.904081 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:21.903966    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="544da439-8c62-47c7-bdb5-256b8b55a890" containerName="revoke-keys"
Oct 30 20:41:21.907510 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:21.907491    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b2ab73-wfflr"
Oct 30 20:41:21.935459 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:21.935428    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rs6g\" (UniqueName: \"kubernetes.io/projected/8e12a218-b48c-46c0-a782-7d7036f83585-kube-api-access-9rs6g\") pod \"maas-api-revoke-sub-e7b2ab73-wfflr\" (UID: \"8e12a218-b48c-46c0-a782-7d7036f83585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b2ab73-wfflr"
Oct 30 20:41:21.935613 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:21.935584    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e12a218-b48c-46c0-a782-7d7036f83585-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7b2ab73-wfflr\" (UID: \"8e12a218-b48c-46c0-a782-7d7036f83585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b2ab73-wfflr"
Oct 30 20:41:21.935708 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:21.935686    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e12a218-b48c-46c0-a782-7d7036f83585-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7b2ab73-wfflr\" (UID: \"8e12a218-b48c-46c0-a782-7d7036f83585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b2ab73-wfflr"
Oct 30 20:41:21.964191 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:21.964152    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 20:41:21.964360 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:21.964216    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:41:21.965416 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:21.965387    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7b2ab73-wfflr"]
Oct 30 20:41:22.037089 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:22.037052    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rs6g\" (UniqueName: \"kubernetes.io/projected/8e12a218-b48c-46c0-a782-7d7036f83585-kube-api-access-9rs6g\") pod \"maas-api-revoke-sub-e7b2ab73-wfflr\" (UID: \"8e12a218-b48c-46c0-a782-7d7036f83585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b2ab73-wfflr"
Oct 30 20:41:22.037284 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:22.037160    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e12a218-b48c-46c0-a782-7d7036f83585-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7b2ab73-wfflr\" (UID: \"8e12a218-b48c-46c0-a782-7d7036f83585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b2ab73-wfflr"
Oct 30 20:41:22.037284 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:22.037225    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e12a218-b48c-46c0-a782-7d7036f83585-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7b2ab73-wfflr\" (UID: \"8e12a218-b48c-46c0-a782-7d7036f83585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b2ab73-wfflr"
Oct 30 20:41:22.038105 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:22.038077    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e12a218-b48c-46c0-a782-7d7036f83585-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7b2ab73-wfflr\" (UID: \"8e12a218-b48c-46c0-a782-7d7036f83585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b2ab73-wfflr"
Oct 30 20:41:22.039927 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:22.039901    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e12a218-b48c-46c0-a782-7d7036f83585-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7b2ab73-wfflr\" (UID: \"8e12a218-b48c-46c0-a782-7d7036f83585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b2ab73-wfflr"
Oct 30 20:41:22.096185 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:22.096157    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rs6g\" (UniqueName: \"kubernetes.io/projected/8e12a218-b48c-46c0-a782-7d7036f83585-kube-api-access-9rs6g\") pod \"maas-api-revoke-sub-e7b2ab73-wfflr\" (UID: \"8e12a218-b48c-46c0-a782-7d7036f83585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b2ab73-wfflr"
Oct 30 20:41:22.220435 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:22.220359    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b2ab73-wfflr"
Oct 30 20:41:22.603195 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:41:22.603165    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e12a218_b48c_46c0_a782_7d7036f83585.slice/crio-efc4d238654b2bf2f8d260bb47765161e03887ea74b5d091736a2b861fc217d2 WatchSource:0}: Error finding container efc4d238654b2bf2f8d260bb47765161e03887ea74b5d091736a2b861fc217d2: Status 404 returned error can't find the container with id efc4d238654b2bf2f8d260bb47765161e03887ea74b5d091736a2b861fc217d2
Oct 30 20:41:22.603750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:22.603727    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7b2ab73-wfflr"]
Oct 30 20:41:22.937382 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:22.937352    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:41:23.078707 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:23.078674    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:23.098369 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:23.098328    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b2ab73-wfflr" event={"ID":"8e12a218-b48c-46c0-a782-7d7036f83585","Type":"ContainerStarted","Data":"b8f5007468b23439952abf0e3549d64ca1161531d491f77aeb0265a56b78b5b2"}
Oct 30 20:41:23.098505 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:23.098385    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b2ab73-wfflr" event={"ID":"8e12a218-b48c-46c0-a782-7d7036f83585","Type":"ContainerStarted","Data":"efc4d238654b2bf2f8d260bb47765161e03887ea74b5d091736a2b861fc217d2"}
Oct 30 20:41:23.220498 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:23.220387    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b2ab73-wfflr" podStartSLOduration=2.220370966 podStartE2EDuration="2.220370966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:23.215534391 +0000 UTC m=+895.132555964" watchObservedRunningTime="2026-10-30 20:41:23.220370966 +0000 UTC m=+895.137392496"
Oct 30 20:41:23.316927 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:23.316884    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:23.847242 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:23.847149    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:24.105477 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:24.105394    5071 generic.go:356] "Generic (PLEG): container finished" podID="8e12a218-b48c-46c0-a782-7d7036f83585" containerID="b8f5007468b23439952abf0e3549d64ca1161531d491f77aeb0265a56b78b5b2" exitCode=0
Oct 30 20:41:24.105896 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:24.105476    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b2ab73-wfflr" event={"ID":"8e12a218-b48c-46c0-a782-7d7036f83585","Type":"ContainerDied","Data":"b8f5007468b23439952abf0e3549d64ca1161531d491f77aeb0265a56b78b5b2"}
Oct 30 20:41:24.278045 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:24.278004    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:25.281251 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:25.281224    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b2ab73-wfflr"
Oct 30 20:41:25.385104 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:25.385022    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e12a218-b48c-46c0-a782-7d7036f83585-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e12a218-b48c-46c0-a782-7d7036f83585-maas-api-xks-ca\") pod \"8e12a218-b48c-46c0-a782-7d7036f83585\" (UID: \"8e12a218-b48c-46c0-a782-7d7036f83585\") "
Oct 30 20:41:25.385104 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:25.385094    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e12a218-b48c-46c0-a782-7d7036f83585-kube-api-access-9rs6g\" (UniqueName: \"kubernetes.io/projected/8e12a218-b48c-46c0-a782-7d7036f83585-kube-api-access-9rs6g\") pod \"8e12a218-b48c-46c0-a782-7d7036f83585\" (UID: \"8e12a218-b48c-46c0-a782-7d7036f83585\") "
Oct 30 20:41:25.385312 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:25.385134    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e12a218-b48c-46c0-a782-7d7036f83585-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e12a218-b48c-46c0-a782-7d7036f83585-maas-api-openshift-ca\") pod \"8e12a218-b48c-46c0-a782-7d7036f83585\" (UID: \"8e12a218-b48c-46c0-a782-7d7036f83585\") "
Oct 30 20:41:25.385549 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:25.385522    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e12a218-b48c-46c0-a782-7d7036f83585-maas-api-openshift-ca" pod "8e12a218-b48c-46c0-a782-7d7036f83585" (UID: "8e12a218-b48c-46c0-a782-7d7036f83585"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:41:25.387164 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:25.387143    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e12a218-b48c-46c0-a782-7d7036f83585-maas-api-xks-ca" pod "8e12a218-b48c-46c0-a782-7d7036f83585" (UID: "8e12a218-b48c-46c0-a782-7d7036f83585"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:41:25.387355 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:25.387327    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e12a218-b48c-46c0-a782-7d7036f83585-kube-api-access-9rs6g" pod "8e12a218-b48c-46c0-a782-7d7036f83585" (UID: "8e12a218-b48c-46c0-a782-7d7036f83585"). InnerVolumeSpecName "kube-api-access-9rs6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:41:25.436738 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:25.436699    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:25.486769 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:25.486738    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e12a218-b48c-46c0-a782-7d7036f83585-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:25.486769 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:25.486766    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rs6g\" (UniqueName: \"kubernetes.io/projected/8e12a218-b48c-46c0-a782-7d7036f83585-kube-api-access-9rs6g\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:25.486968 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:25.486781    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e12a218-b48c-46c0-a782-7d7036f83585-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:26.118987 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:26.118951    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b2ab73-wfflr" event={"ID":"8e12a218-b48c-46c0-a782-7d7036f83585","Type":"ContainerDied","Data":"efc4d238654b2bf2f8d260bb47765161e03887ea74b5d091736a2b861fc217d2"}
Oct 30 20:41:26.118987 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:26.118978    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b2ab73-wfflr"
Oct 30 20:41:26.119200 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:26.119003    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efc4d238654b2bf2f8d260bb47765161e03887ea74b5d091736a2b861fc217d2"
Oct 30 20:41:27.402991 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:27.402946    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:28.318733 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:28.318707    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8qvb_91fe9e06-0c30-4000-bddb-2ba89cd048cc/console-operator/2.log"
Oct 30 20:41:28.318926 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:28.318709    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8qvb_91fe9e06-0c30-4000-bddb-2ba89cd048cc/console-operator/2.log"
Oct 30 20:41:28.326396 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:28.326376    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q4f4h_80cae79e-b60f-4efe-bb58-e77753a457e5/ovn-acl-logging/0.log"
Oct 30 20:41:28.326513 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:28.326497    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q4f4h_80cae79e-b60f-4efe-bb58-e77753a457e5/ovn-acl-logging/0.log"
Oct 30 20:41:28.950596 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:28.950549    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:32.032388 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:32.032353    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"]
Oct 30 20:41:32.032953 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:32.032555    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7" podUID="1326fe90-f3ab-4922-8ab9-a6eab1dcf665" containerName="istio-proxy" containerID="cri-o://c127aec28f417408eb2ad964c94910dec03e8f8ae6778b2720b79dd0a3951de9" gracePeriod=30
Oct 30 20:41:33.986101 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:33.986062    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:37.194734 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.194692    5071 generic.go:356] "Generic (PLEG): container finished" podID="1326fe90-f3ab-4922-8ab9-a6eab1dcf665" containerID="c127aec28f417408eb2ad964c94910dec03e8f8ae6778b2720b79dd0a3951de9" exitCode=0
Oct 30 20:41:37.195157 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.194756    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7" event={"ID":"1326fe90-f3ab-4922-8ab9-a6eab1dcf665","Type":"ContainerDied","Data":"c127aec28f417408eb2ad964c94910dec03e8f8ae6778b2720b79dd0a3951de9"}
Oct 30 20:41:37.195157 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.194858    5071 scope.go:122] "RemoveContainer" containerID="6f4b4ac41dc3c538c0bc81f7f4833c35ab5de48c80a76db4e1ed45a70038a4a1"
Oct 30 20:41:37.318196 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.318137    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:41:37.420978 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.420950    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-istio-data\") pod \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\" (UID: \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\") "
Oct 30 20:41:37.421177 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.421000    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-workload-certs\") pod \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\" (UID: \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\") "
Oct 30 20:41:37.421177 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.421026    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-istio-podinfo\") pod \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\" (UID: \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\") "
Oct 30 20:41:37.421177 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.421072    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-istio-envoy\") pod \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\" (UID: \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\") "
Oct 30 20:41:37.421177 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.421135    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-istio-token\" (UniqueName: \"kubernetes.io/projected/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-istio-token\") pod \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\" (UID: \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\") "
Oct 30 20:41:37.421402 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.421189    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-kube-api-access-nkp5w\" (UniqueName: \"kubernetes.io/projected/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-kube-api-access-nkp5w\") pod \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\" (UID: \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\") "
Oct 30 20:41:37.421402 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.421230    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-credential-socket\") pod \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\" (UID: \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\") "
Oct 30 20:41:37.421402 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.421233    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-workload-certs" pod "1326fe90-f3ab-4922-8ab9-a6eab1dcf665" (UID: "1326fe90-f3ab-4922-8ab9-a6eab1dcf665"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:41:37.421402 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.421265    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-istiod-ca-cert\") pod \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\" (UID: \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\") "
Oct 30 20:41:37.421402 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.421270    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-istio-data" pod "1326fe90-f3ab-4922-8ab9-a6eab1dcf665" (UID: "1326fe90-f3ab-4922-8ab9-a6eab1dcf665"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:41:37.421402 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.421321    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-workload-socket\") pod \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\" (UID: \"1326fe90-f3ab-4922-8ab9-a6eab1dcf665\") "
Oct 30 20:41:37.421761 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.421505    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-credential-socket" pod "1326fe90-f3ab-4922-8ab9-a6eab1dcf665" (UID: "1326fe90-f3ab-4922-8ab9-a6eab1dcf665"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:41:37.421761 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.421706    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-workload-socket" pod "1326fe90-f3ab-4922-8ab9-a6eab1dcf665" (UID: "1326fe90-f3ab-4922-8ab9-a6eab1dcf665"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:41:37.421761 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.421739    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-istiod-ca-cert" pod "1326fe90-f3ab-4922-8ab9-a6eab1dcf665" (UID: "1326fe90-f3ab-4922-8ab9-a6eab1dcf665"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:41:37.421929 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.421881    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:37.421929 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.421903    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:37.422065 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.421994    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:37.422106 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.422094    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:37.422140 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.422112    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:37.423722 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.423694    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-istio-envoy" pod "1326fe90-f3ab-4922-8ab9-a6eab1dcf665" (UID: "1326fe90-f3ab-4922-8ab9-a6eab1dcf665"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:41:37.423857 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.423720    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-istio-podinfo" pod "1326fe90-f3ab-4922-8ab9-a6eab1dcf665" (UID: "1326fe90-f3ab-4922-8ab9-a6eab1dcf665"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:41:37.424053 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.424030    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-kube-api-access-nkp5w" pod "1326fe90-f3ab-4922-8ab9-a6eab1dcf665" (UID: "1326fe90-f3ab-4922-8ab9-a6eab1dcf665"). InnerVolumeSpecName "kube-api-access-nkp5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:41:37.424053 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.424044    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-istio-token" pod "1326fe90-f3ab-4922-8ab9-a6eab1dcf665" (UID: "1326fe90-f3ab-4922-8ab9-a6eab1dcf665"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:41:37.440410 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.440367    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-640e29ac-57vxk"]
Oct 30 20:41:37.441183 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.441164    5071 cpu_manager.go:414] "containerMap: removing container" podUID="1326fe90-f3ab-4922-8ab9-a6eab1dcf665" containerName="istio-proxy"
Oct 30 20:41:37.441275 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.441184    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1326fe90-f3ab-4922-8ab9-a6eab1dcf665" containerName="istio-proxy"
Oct 30 20:41:37.441275 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.441205    5071 cpu_manager.go:414] "containerMap: removing container" podUID="8e12a218-b48c-46c0-a782-7d7036f83585" containerName="revoke-keys"
Oct 30 20:41:37.441275 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.441213    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e12a218-b48c-46c0-a782-7d7036f83585" containerName="revoke-keys"
Oct 30 20:41:37.441275 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.441226    5071 cpu_manager.go:414] "containerMap: removing container" podUID="1326fe90-f3ab-4922-8ab9-a6eab1dcf665" containerName="istio-proxy"
Oct 30 20:41:37.441275 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.441232    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1326fe90-f3ab-4922-8ab9-a6eab1dcf665" containerName="istio-proxy"
Oct 30 20:41:37.441538 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.441370    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e12a218-b48c-46c0-a782-7d7036f83585" containerName="revoke-keys"
Oct 30 20:41:37.441538 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.441381    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="1326fe90-f3ab-4922-8ab9-a6eab1dcf665" containerName="istio-proxy"
Oct 30 20:41:37.441538 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.441399    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="1326fe90-f3ab-4922-8ab9-a6eab1dcf665" containerName="istio-proxy"
Oct 30 20:41:37.445584 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.445563    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-640e29ac-57vxk"
Oct 30 20:41:37.523128 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.523094    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6280eea7-d203-4c65-883d-a8027a1984da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-640e29ac-57vxk\" (UID: \"6280eea7-d203-4c65-883d-a8027a1984da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-640e29ac-57vxk"
Oct 30 20:41:37.523128 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.523127    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjqpt\" (UniqueName: \"kubernetes.io/projected/6280eea7-d203-4c65-883d-a8027a1984da-kube-api-access-cjqpt\") pod \"maas-api-revoke-sub-640e29ac-57vxk\" (UID: \"6280eea7-d203-4c65-883d-a8027a1984da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-640e29ac-57vxk"
Oct 30 20:41:37.523339 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.523270    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6280eea7-d203-4c65-883d-a8027a1984da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-640e29ac-57vxk\" (UID: \"6280eea7-d203-4c65-883d-a8027a1984da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-640e29ac-57vxk"
Oct 30 20:41:37.523406 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.523383    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:37.523406 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.523400    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:37.523490 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.523411    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkp5w\" (UniqueName: \"kubernetes.io/projected/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-kube-api-access-nkp5w\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:37.523490 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.523421    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1326fe90-f3ab-4922-8ab9-a6eab1dcf665-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:37.544396 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.544365    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:41:37.544567 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.544409    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 20:41:37.551450 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.551420    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-640e29ac-57vxk"]
Oct 30 20:41:37.624362 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.624276    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6280eea7-d203-4c65-883d-a8027a1984da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-640e29ac-57vxk\" (UID: \"6280eea7-d203-4c65-883d-a8027a1984da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-640e29ac-57vxk"
Oct 30 20:41:37.624511 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.624395    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6280eea7-d203-4c65-883d-a8027a1984da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-640e29ac-57vxk\" (UID: \"6280eea7-d203-4c65-883d-a8027a1984da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-640e29ac-57vxk"
Oct 30 20:41:37.624511 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.624415    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjqpt\" (UniqueName: \"kubernetes.io/projected/6280eea7-d203-4c65-883d-a8027a1984da-kube-api-access-cjqpt\") pod \"maas-api-revoke-sub-640e29ac-57vxk\" (UID: \"6280eea7-d203-4c65-883d-a8027a1984da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-640e29ac-57vxk"
Oct 30 20:41:37.625141 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.625114    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6280eea7-d203-4c65-883d-a8027a1984da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-640e29ac-57vxk\" (UID: \"6280eea7-d203-4c65-883d-a8027a1984da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-640e29ac-57vxk"
Oct 30 20:41:37.627098 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.627072    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6280eea7-d203-4c65-883d-a8027a1984da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-640e29ac-57vxk\" (UID: \"6280eea7-d203-4c65-883d-a8027a1984da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-640e29ac-57vxk"
Oct 30 20:41:37.684065 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.684028    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjqpt\" (UniqueName: \"kubernetes.io/projected/6280eea7-d203-4c65-883d-a8027a1984da-kube-api-access-cjqpt\") pod \"maas-api-revoke-sub-640e29ac-57vxk\" (UID: \"6280eea7-d203-4c65-883d-a8027a1984da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-640e29ac-57vxk"
Oct 30 20:41:37.759303 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:37.759266    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-640e29ac-57vxk"
Oct 30 20:41:38.136333 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:38.136300    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-640e29ac-57vxk"]
Oct 30 20:41:38.138317 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:41:38.138289    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6280eea7_d203_4c65_883d_a8027a1984da.slice/crio-61cb2a4ddd4933cf2b7826bfe0582decae36ee69cdc6bd9abd191fd593330f50 WatchSource:0}: Error finding container 61cb2a4ddd4933cf2b7826bfe0582decae36ee69cdc6bd9abd191fd593330f50: Status 404 returned error can't find the container with id 61cb2a4ddd4933cf2b7826bfe0582decae36ee69cdc6bd9abd191fd593330f50
Oct 30 20:41:38.203373 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:38.203337    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-640e29ac-57vxk" event={"ID":"6280eea7-d203-4c65-883d-a8027a1984da","Type":"ContainerStarted","Data":"61cb2a4ddd4933cf2b7826bfe0582decae36ee69cdc6bd9abd191fd593330f50"}
Oct 30 20:41:38.204209 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:38.204179    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7" event={"ID":"1326fe90-f3ab-4922-8ab9-a6eab1dcf665","Type":"ContainerDied","Data":"3950094a7119ca1da065f69c71d7131d56772cd5dfda9bd2000b26cee5990ec9"}
Oct 30 20:41:38.204323 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:38.204225    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"
Oct 30 20:41:38.204323 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:38.204239    5071 scope.go:122] "RemoveContainer" containerID="c127aec28f417408eb2ad964c94910dec03e8f8ae6778b2720b79dd0a3951de9"
Oct 30 20:41:38.378790 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:38.378763    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"]
Oct 30 20:41:38.428742 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:38.428659    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-9lhz7"]
Oct 30 20:41:39.211978 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:39.211948    5071 generic.go:356] "Generic (PLEG): container finished" podID="6280eea7-d203-4c65-883d-a8027a1984da" containerID="83d2bfbe5d1608390995c005672596c1eb2dff11e4c00924166ae63d23ad3af8" exitCode=0
Oct 30 20:41:39.212465 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:39.212020    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-640e29ac-57vxk" event={"ID":"6280eea7-d203-4c65-883d-a8027a1984da","Type":"ContainerDied","Data":"83d2bfbe5d1608390995c005672596c1eb2dff11e4c00924166ae63d23ad3af8"}
Oct 30 20:41:39.265072 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:39.264986    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:40.201280 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:40.201250    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1326fe90-f3ab-4922-8ab9-a6eab1dcf665" path="/var/lib/kubelet/pods/1326fe90-f3ab-4922-8ab9-a6eab1dcf665/volumes"
Oct 30 20:41:40.388856 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:40.388806    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-640e29ac-57vxk"
Oct 30 20:41:40.553971 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:40.553943    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6280eea7-d203-4c65-883d-a8027a1984da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6280eea7-d203-4c65-883d-a8027a1984da-maas-api-xks-ca\") pod \"6280eea7-d203-4c65-883d-a8027a1984da\" (UID: \"6280eea7-d203-4c65-883d-a8027a1984da\") "
Oct 30 20:41:40.554150 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:40.554011    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6280eea7-d203-4c65-883d-a8027a1984da-kube-api-access-cjqpt\" (UniqueName: \"kubernetes.io/projected/6280eea7-d203-4c65-883d-a8027a1984da-kube-api-access-cjqpt\") pod \"6280eea7-d203-4c65-883d-a8027a1984da\" (UID: \"6280eea7-d203-4c65-883d-a8027a1984da\") "
Oct 30 20:41:40.554216 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:40.554160    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6280eea7-d203-4c65-883d-a8027a1984da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6280eea7-d203-4c65-883d-a8027a1984da-maas-api-openshift-ca\") pod \"6280eea7-d203-4c65-883d-a8027a1984da\" (UID: \"6280eea7-d203-4c65-883d-a8027a1984da\") "
Oct 30 20:41:40.554492 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:40.554467    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6280eea7-d203-4c65-883d-a8027a1984da-maas-api-openshift-ca" pod "6280eea7-d203-4c65-883d-a8027a1984da" (UID: "6280eea7-d203-4c65-883d-a8027a1984da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:41:40.556393 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:40.556368    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6280eea7-d203-4c65-883d-a8027a1984da-maas-api-xks-ca" pod "6280eea7-d203-4c65-883d-a8027a1984da" (UID: "6280eea7-d203-4c65-883d-a8027a1984da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:41:40.556478 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:40.556368    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6280eea7-d203-4c65-883d-a8027a1984da-kube-api-access-cjqpt" pod "6280eea7-d203-4c65-883d-a8027a1984da" (UID: "6280eea7-d203-4c65-883d-a8027a1984da"). InnerVolumeSpecName "kube-api-access-cjqpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:41:40.655714 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:40.655639    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjqpt\" (UniqueName: \"kubernetes.io/projected/6280eea7-d203-4c65-883d-a8027a1984da-kube-api-access-cjqpt\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:40.655910 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:40.655728    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6280eea7-d203-4c65-883d-a8027a1984da-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:40.655910 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:40.655749    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6280eea7-d203-4c65-883d-a8027a1984da-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:41.227569 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:41.227538    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-640e29ac-57vxk"
Oct 30 20:41:41.227752 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:41.227562    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-640e29ac-57vxk" event={"ID":"6280eea7-d203-4c65-883d-a8027a1984da","Type":"ContainerDied","Data":"61cb2a4ddd4933cf2b7826bfe0582decae36ee69cdc6bd9abd191fd593330f50"}
Oct 30 20:41:41.227752 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:41.227619    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61cb2a4ddd4933cf2b7826bfe0582decae36ee69cdc6bd9abd191fd593330f50"
Oct 30 20:41:43.251772 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:43.251739    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"]
Oct 30 20:41:43.252601 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:43.252578    5071 cpu_manager.go:414] "containerMap: removing container" podUID="6280eea7-d203-4c65-883d-a8027a1984da" containerName="revoke-keys"
Oct 30 20:41:43.252638 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:43.252606    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6280eea7-d203-4c65-883d-a8027a1984da" containerName="revoke-keys"
Oct 30 20:41:43.252811 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:43.252798    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="6280eea7-d203-4c65-883d-a8027a1984da" containerName="revoke-keys"
Oct 30 20:41:43.256661 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:43.256642    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"
Oct 30 20:41:43.312006 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:43.311978    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-395022-openshift-default-dockercfg-7nmw8\""
Oct 30 20:41:43.365792 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:43.365753    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"]
Oct 30 20:41:43.385186 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:43.385159    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b1175c-ea5d-49a8-b721-93fd709970c1-credential-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-8g82t\" (UID: \"00b1175c-ea5d-49a8-b721-93fd709970c1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"
Oct 30 20:41:43.385358 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:43.385211    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00b1175c-ea5d-49a8-b721-93fd709970c1-istiod-ca-cert\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-8g82t\" (UID: \"00b1175c-ea5d-49a8-b721-93fd709970c1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"
Oct 30 20:41:43.385358 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:43.385239    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00b1175c-ea5d-49a8-b721-93fd709970c1-istio-envoy\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-8g82t\" (UID: \"00b1175c-ea5d-49a8-b721-93fd709970c1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"
Oct 30 20:41:43.385447 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:43.385374    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00b1175c-ea5d-49a8-b721-93fd709970c1-istio-token\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-8g82t\" (UID: \"00b1175c-ea5d-49a8-b721-93fd709970c1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"
Oct 30 20:41:43.385484 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:43.385441    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00b1175c-ea5d-49a8-b721-93fd709970c1-workload-certs\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-8g82t\" (UID: \"00b1175c-ea5d-49a8-b721-93fd709970c1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"
Oct 30 20:41:43.385484 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:43.385478    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b1175c-ea5d-49a8-b721-93fd709970c1-workload-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-8g82t\" (UID: \"00b1175c-ea5d-49a8-b721-93fd709970c1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"
Oct 30 20:41:43.385578 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:43.385540    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00b1175c-ea5d-49a8-b721-93fd709970c1-istio-podinfo\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-8g82t\" (UID: \"00b1175c-ea5d-49a8-b721-93fd709970c1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"
Oct 30 20:41:43.385578 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:43.385567    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00b1175c-ea5d-49a8-b721-93fd709970c1-istio-data\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-8g82t\" (UID: \"00b1175c-ea5d-49a8-b721-93fd709970c1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"
Oct 30 20:41:43.385677 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:43.385582    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh8fl\" (UniqueName: \"kubernetes.io/projected/00b1175c-ea5d-49a8-b721-93fd709970c1-kube-api-access-vh8fl\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-8g82t\" (UID: \"00b1175c-ea5d-49a8-b721-93fd709970c1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"
Oct 30 20:41:43.486713 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:43.486678    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b1175c-ea5d-49a8-b721-93fd709970c1-credential-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-8g82t\" (UID: \"00b1175c-ea5d-49a8-b721-93fd709970c1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"
Oct 30 20:41:43.486915 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:43.486732    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00b1175c-ea5d-49a8-b721-93fd709970c1-istiod-ca-cert\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-8g82t\" (UID: \"00b1175c-ea5d-49a8-b721-93fd709970c1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"
Oct 30 20:41:43.486915 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:43.486754    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00b1175c-ea5d-49a8-b721-93fd709970c1-istio-envoy\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-8g82t\" (UID: \"00b1175c-ea5d-49a8-b721-93fd709970c1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"
Oct 30 20:41:43.486915 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:43.486781    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00b1175c-ea5d-49a8-b721-93fd709970c1-istio-token\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-8g82t\" (UID: \"00b1175c-ea5d-49a8-b721-93fd709970c1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"
Oct 30 20:41:43.486915 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:43.486842    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00b1175c-ea5d-49a8-b721-93fd709970c1-workload-certs\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-8g82t\" (UID: \"00b1175c-ea5d-49a8-b721-93fd709970c1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"
Oct 30 20:41:43.486915 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:43.486891    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b1175c-ea5d-49a8-b721-93fd709970c1-workload-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-8g82t\" (UID: \"00b1175c-ea5d-49a8-b721-93fd709970c1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"
Oct 30 20:41:43.487174 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:43.486938    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00b1175c-ea5d-49a8-b721-93fd709970c1-istio-podinfo\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-8g82t\" (UID: \"00b1175c-ea5d-49a8-b721-93fd709970c1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"
Oct 30 20:41:43.487174 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:43.487057    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00b1175c-ea5d-49a8-b721-93fd709970c1-istio-data\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-8g82t\" (UID: \"00b1175c-ea5d-49a8-b721-93fd709970c1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"
Oct 30 20:41:43.487174 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:43.487103    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh8fl\" (UniqueName: \"kubernetes.io/projected/00b1175c-ea5d-49a8-b721-93fd709970c1-kube-api-access-vh8fl\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-8g82t\" (UID: \"00b1175c-ea5d-49a8-b721-93fd709970c1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"
Oct 30 20:41:43.487174 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:43.487163    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b1175c-ea5d-49a8-b721-93fd709970c1-credential-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-8g82t\" (UID: \"00b1175c-ea5d-49a8-b721-93fd709970c1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"
Oct 30 20:41:43.487358 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:43.487206    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00b1175c-ea5d-49a8-b721-93fd709970c1-workload-certs\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-8g82t\" (UID: \"00b1175c-ea5d-49a8-b721-93fd709970c1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"
Oct 30 20:41:43.487491 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:43.487466    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00b1175c-ea5d-49a8-b721-93fd709970c1-istiod-ca-cert\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-8g82t\" (UID: \"00b1175c-ea5d-49a8-b721-93fd709970c1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"
Oct 30 20:41:43.487729 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:43.487706    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b1175c-ea5d-49a8-b721-93fd709970c1-workload-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-8g82t\" (UID: \"00b1175c-ea5d-49a8-b721-93fd709970c1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"
Oct 30 20:41:43.487887 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:43.487868    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00b1175c-ea5d-49a8-b721-93fd709970c1-istio-data\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-8g82t\" (UID: \"00b1175c-ea5d-49a8-b721-93fd709970c1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"
Oct 30 20:41:43.489155 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:43.489137    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00b1175c-ea5d-49a8-b721-93fd709970c1-istio-envoy\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-8g82t\" (UID: \"00b1175c-ea5d-49a8-b721-93fd709970c1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"
Oct 30 20:41:43.489369 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:43.489352    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00b1175c-ea5d-49a8-b721-93fd709970c1-istio-podinfo\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-8g82t\" (UID: \"00b1175c-ea5d-49a8-b721-93fd709970c1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"
Oct 30 20:41:43.547278 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:43.547245    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00b1175c-ea5d-49a8-b721-93fd709970c1-istio-token\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-8g82t\" (UID: \"00b1175c-ea5d-49a8-b721-93fd709970c1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"
Oct 30 20:41:43.547464 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:43.547358    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh8fl\" (UniqueName: \"kubernetes.io/projected/00b1175c-ea5d-49a8-b721-93fd709970c1-kube-api-access-vh8fl\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-8g82t\" (UID: \"00b1175c-ea5d-49a8-b721-93fd709970c1\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"
Oct 30 20:41:43.580073 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:43.580032    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"
Oct 30 20:41:44.016405 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:44.016375    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"]
Oct 30 20:41:44.017202 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:41:44.017176    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00b1175c_ea5d_49a8_b721_93fd709970c1.slice/crio-0ce2edc5e2b46e3a69d4bb6a681ea8cf0d030ed5f99558d59f2623e3cb8175c8 WatchSource:0}: Error finding container 0ce2edc5e2b46e3a69d4bb6a681ea8cf0d030ed5f99558d59f2623e3cb8175c8: Status 404 returned error can't find the container with id 0ce2edc5e2b46e3a69d4bb6a681ea8cf0d030ed5f99558d59f2623e3cb8175c8
Oct 30 20:41:44.019492 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:44.019461    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:41:44.019570 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:44.019521    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:41:44.019570 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:44.019552    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:41:44.232501 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:44.230885    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:44.255790 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:44.255744    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t" event={"ID":"00b1175c-ea5d-49a8-b721-93fd709970c1","Type":"ContainerStarted","Data":"15ede708ebc06744572625fcfd27ff799d5c51483946b49ccf1e095a9db1eab4"}
Oct 30 20:41:44.256231 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:44.255812    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t" event={"ID":"00b1175c-ea5d-49a8-b721-93fd709970c1","Type":"ContainerStarted","Data":"0ce2edc5e2b46e3a69d4bb6a681ea8cf0d030ed5f99558d59f2623e3cb8175c8"}
Oct 30 20:41:44.385631 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:44.385559    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t" podStartSLOduration=1.385539336 podStartE2EDuration="1.385539336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:44.384570204 +0000 UTC m=+916.301591741" watchObservedRunningTime="2026-10-30 20:41:44.385539336 +0000 UTC m=+916.302560874"
Oct 30 20:41:44.580248 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:44.580213    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"
Oct 30 20:41:45.582152 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:45.582121    5071 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 20:41:45.582683 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:45.582172    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t" podUID="00b1175c-ea5d-49a8-b721-93fd709970c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 20:41:46.580974 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:46.580936    5071 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 20:41:46.581170 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:46.580998    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t" podUID="00b1175c-ea5d-49a8-b721-93fd709970c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 20:41:46.753775 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:41:46.753672    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:41:47.580325 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:47.580293    5071 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 20:41:47.580495 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:47.580352    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t" podUID="00b1175c-ea5d-49a8-b721-93fd709970c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 20:41:48.307336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:48.307301    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdb77bd8-x6tld"]
Oct 30 20:41:48.311589 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:48.311571    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb77bd8-x6tld"
Oct 30 20:41:48.374243 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:48.374215    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:41:48.416223 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:48.416180    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 20:41:48.419155 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:48.419129    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdb77bd8-x6tld"]
Oct 30 20:41:48.478361 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:48.478332    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f376fda3-4de7-4c54-a64a-e2af38ecb340-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdb77bd8-x6tld\" (UID: \"f376fda3-4de7-4c54-a64a-e2af38ecb340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb77bd8-x6tld"
Oct 30 20:41:48.478537 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:48.478385    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f376fda3-4de7-4c54-a64a-e2af38ecb340-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdb77bd8-x6tld\" (UID: \"f376fda3-4de7-4c54-a64a-e2af38ecb340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb77bd8-x6tld"
Oct 30 20:41:48.478537 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:48.478465    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9xkm\" (UniqueName: \"kubernetes.io/projected/f376fda3-4de7-4c54-a64a-e2af38ecb340-kube-api-access-j9xkm\") pod \"maas-api-revoke-sub-cdb77bd8-x6tld\" (UID: \"f376fda3-4de7-4c54-a64a-e2af38ecb340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb77bd8-x6tld"
Oct 30 20:41:48.579755 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:48.579666    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f376fda3-4de7-4c54-a64a-e2af38ecb340-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdb77bd8-x6tld\" (UID: \"f376fda3-4de7-4c54-a64a-e2af38ecb340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb77bd8-x6tld"
Oct 30 20:41:48.579755 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:48.579715    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9xkm\" (UniqueName: \"kubernetes.io/projected/f376fda3-4de7-4c54-a64a-e2af38ecb340-kube-api-access-j9xkm\") pod \"maas-api-revoke-sub-cdb77bd8-x6tld\" (UID: \"f376fda3-4de7-4c54-a64a-e2af38ecb340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb77bd8-x6tld"
Oct 30 20:41:48.579986 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:48.579865    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f376fda3-4de7-4c54-a64a-e2af38ecb340-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdb77bd8-x6tld\" (UID: \"f376fda3-4de7-4c54-a64a-e2af38ecb340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb77bd8-x6tld"
Oct 30 20:41:48.580335 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:48.580303    5071 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 20:41:48.580460 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:48.580358    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t" podUID="00b1175c-ea5d-49a8-b721-93fd709970c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 20:41:48.580460 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:48.580388    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f376fda3-4de7-4c54-a64a-e2af38ecb340-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdb77bd8-x6tld\" (UID: \"f376fda3-4de7-4c54-a64a-e2af38ecb340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb77bd8-x6tld"
Oct 30 20:41:48.582386 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:48.582364    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f376fda3-4de7-4c54-a64a-e2af38ecb340-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdb77bd8-x6tld\" (UID: \"f376fda3-4de7-4c54-a64a-e2af38ecb340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb77bd8-x6tld"
Oct 30 20:41:48.654355 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:48.654316    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9xkm\" (UniqueName: \"kubernetes.io/projected/f376fda3-4de7-4c54-a64a-e2af38ecb340-kube-api-access-j9xkm\") pod \"maas-api-revoke-sub-cdb77bd8-x6tld\" (UID: \"f376fda3-4de7-4c54-a64a-e2af38ecb340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb77bd8-x6tld"
Oct 30 20:41:48.925471 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:48.925391    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb77bd8-x6tld"
Oct 30 20:41:49.358846 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:49.357998    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdb77bd8-x6tld"]
Oct 30 20:41:49.515931 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:49.515890    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:49.584894 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:49.584631    5071 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 30 20:41:49.584894 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:49.584703    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t" podUID="00b1175c-ea5d-49a8-b721-93fd709970c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 30 20:41:50.322811 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:50.322772    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb77bd8-x6tld" event={"ID":"f376fda3-4de7-4c54-a64a-e2af38ecb340","Type":"ContainerStarted","Data":"1b37f262855ab980b0f69a6699dc2da69eabdb11986b08dd68a3b701c5f0247f"}
Oct 30 20:41:50.323014 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:50.322845    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb77bd8-x6tld" event={"ID":"f376fda3-4de7-4c54-a64a-e2af38ecb340","Type":"ContainerStarted","Data":"eb69c62762737f93d534a8752561c096d65d316fb0189c3f366948576614410c"}
Oct 30 20:41:50.741801 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:50.741717    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"
Oct 30 20:41:50.742289 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:50.742116    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"
Oct 30 20:41:50.743086 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:50.743064    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"
Oct 30 20:41:50.860795 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:50.860730    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb77bd8-x6tld" podStartSLOduration=2.860713144 podStartE2EDuration="2.860713144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:50.440448804 +0000 UTC m=+922.357470342" watchObservedRunningTime="2026-10-30 20:41:50.860713144 +0000 UTC m=+922.777734692"
Oct 30 20:41:51.330438 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:51.330399    5071 generic.go:356] "Generic (PLEG): container finished" podID="f376fda3-4de7-4c54-a64a-e2af38ecb340" containerID="1b37f262855ab980b0f69a6699dc2da69eabdb11986b08dd68a3b701c5f0247f" exitCode=0
Oct 30 20:41:51.330636 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:51.330489    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb77bd8-x6tld" event={"ID":"f376fda3-4de7-4c54-a64a-e2af38ecb340","Type":"ContainerDied","Data":"1b37f262855ab980b0f69a6699dc2da69eabdb11986b08dd68a3b701c5f0247f"}
Oct 30 20:41:52.517139 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:52.517113    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb77bd8-x6tld"
Oct 30 20:41:52.623324 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:52.623245    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f376fda3-4de7-4c54-a64a-e2af38ecb340-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f376fda3-4de7-4c54-a64a-e2af38ecb340-maas-api-openshift-ca\") pod \"f376fda3-4de7-4c54-a64a-e2af38ecb340\" (UID: \"f376fda3-4de7-4c54-a64a-e2af38ecb340\") "
Oct 30 20:41:52.623465 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:52.623335    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f376fda3-4de7-4c54-a64a-e2af38ecb340-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f376fda3-4de7-4c54-a64a-e2af38ecb340-maas-api-xks-ca\") pod \"f376fda3-4de7-4c54-a64a-e2af38ecb340\" (UID: \"f376fda3-4de7-4c54-a64a-e2af38ecb340\") "
Oct 30 20:41:52.623465 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:52.623378    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f376fda3-4de7-4c54-a64a-e2af38ecb340-kube-api-access-j9xkm\" (UniqueName: \"kubernetes.io/projected/f376fda3-4de7-4c54-a64a-e2af38ecb340-kube-api-access-j9xkm\") pod \"f376fda3-4de7-4c54-a64a-e2af38ecb340\" (UID: \"f376fda3-4de7-4c54-a64a-e2af38ecb340\") "
Oct 30 20:41:52.623616 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:52.623591    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f376fda3-4de7-4c54-a64a-e2af38ecb340-maas-api-openshift-ca" pod "f376fda3-4de7-4c54-a64a-e2af38ecb340" (UID: "f376fda3-4de7-4c54-a64a-e2af38ecb340"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:41:52.623700 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:52.623685    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f376fda3-4de7-4c54-a64a-e2af38ecb340-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:52.625686 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:52.625657    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f376fda3-4de7-4c54-a64a-e2af38ecb340-maas-api-xks-ca" pod "f376fda3-4de7-4c54-a64a-e2af38ecb340" (UID: "f376fda3-4de7-4c54-a64a-e2af38ecb340"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:41:52.625686 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:52.625684    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f376fda3-4de7-4c54-a64a-e2af38ecb340-kube-api-access-j9xkm" pod "f376fda3-4de7-4c54-a64a-e2af38ecb340" (UID: "f376fda3-4de7-4c54-a64a-e2af38ecb340"). InnerVolumeSpecName "kube-api-access-j9xkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:41:52.724710 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:52.724681    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f376fda3-4de7-4c54-a64a-e2af38ecb340-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:52.724710 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:52.724710    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9xkm\" (UniqueName: \"kubernetes.io/projected/f376fda3-4de7-4c54-a64a-e2af38ecb340-kube-api-access-j9xkm\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:52.855335 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:52.855297    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e580a5-ftxs5"]
Oct 30 20:41:52.855958 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:52.855937    5071 cpu_manager.go:414] "containerMap: removing container" podUID="f376fda3-4de7-4c54-a64a-e2af38ecb340" containerName="revoke-keys"
Oct 30 20:41:52.855958 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:52.855958    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f376fda3-4de7-4c54-a64a-e2af38ecb340" containerName="revoke-keys"
Oct 30 20:41:52.856175 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:52.856138    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="f376fda3-4de7-4c54-a64a-e2af38ecb340" containerName="revoke-keys"
Oct 30 20:41:52.859575 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:52.859557    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e580a5-ftxs5"
Oct 30 20:41:52.918379 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:52.918290    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e580a5-ftxs5"]
Oct 30 20:41:52.927737 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:52.927706    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e936350a-b13c-45e6-ac5a-8d2097b9bba0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28e580a5-ftxs5\" (UID: \"e936350a-b13c-45e6-ac5a-8d2097b9bba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e580a5-ftxs5"
Oct 30 20:41:52.927900 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:52.927793    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e936350a-b13c-45e6-ac5a-8d2097b9bba0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28e580a5-ftxs5\" (UID: \"e936350a-b13c-45e6-ac5a-8d2097b9bba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e580a5-ftxs5"
Oct 30 20:41:52.928014 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:52.927991    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb2fc\" (UniqueName: \"kubernetes.io/projected/e936350a-b13c-45e6-ac5a-8d2097b9bba0-kube-api-access-nb2fc\") pod \"maas-api-revoke-sub-28e580a5-ftxs5\" (UID: \"e936350a-b13c-45e6-ac5a-8d2097b9bba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e580a5-ftxs5"
Oct 30 20:41:53.029412 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:53.029378    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e936350a-b13c-45e6-ac5a-8d2097b9bba0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28e580a5-ftxs5\" (UID: \"e936350a-b13c-45e6-ac5a-8d2097b9bba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e580a5-ftxs5"
Oct 30 20:41:53.029607 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:53.029425    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e936350a-b13c-45e6-ac5a-8d2097b9bba0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28e580a5-ftxs5\" (UID: \"e936350a-b13c-45e6-ac5a-8d2097b9bba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e580a5-ftxs5"
Oct 30 20:41:53.029607 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:53.029486    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb2fc\" (UniqueName: \"kubernetes.io/projected/e936350a-b13c-45e6-ac5a-8d2097b9bba0-kube-api-access-nb2fc\") pod \"maas-api-revoke-sub-28e580a5-ftxs5\" (UID: \"e936350a-b13c-45e6-ac5a-8d2097b9bba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e580a5-ftxs5"
Oct 30 20:41:53.030238 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:53.030208    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e936350a-b13c-45e6-ac5a-8d2097b9bba0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28e580a5-ftxs5\" (UID: \"e936350a-b13c-45e6-ac5a-8d2097b9bba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e580a5-ftxs5"
Oct 30 20:41:53.032162 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:53.032140    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e936350a-b13c-45e6-ac5a-8d2097b9bba0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28e580a5-ftxs5\" (UID: \"e936350a-b13c-45e6-ac5a-8d2097b9bba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e580a5-ftxs5"
Oct 30 20:41:53.088808 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:53.088774    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb2fc\" (UniqueName: \"kubernetes.io/projected/e936350a-b13c-45e6-ac5a-8d2097b9bba0-kube-api-access-nb2fc\") pod \"maas-api-revoke-sub-28e580a5-ftxs5\" (UID: \"e936350a-b13c-45e6-ac5a-8d2097b9bba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e580a5-ftxs5"
Oct 30 20:41:53.173598 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:53.173516    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e580a5-ftxs5"
Oct 30 20:41:53.345632 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:53.345597    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb77bd8-x6tld"
Oct 30 20:41:53.345813 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:53.345608    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb77bd8-x6tld" event={"ID":"f376fda3-4de7-4c54-a64a-e2af38ecb340","Type":"ContainerDied","Data":"eb69c62762737f93d534a8752561c096d65d316fb0189c3f366948576614410c"}
Oct 30 20:41:53.345813 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:53.345719    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb69c62762737f93d534a8752561c096d65d316fb0189c3f366948576614410c"
Oct 30 20:41:53.559167 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:53.559138    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e580a5-ftxs5"]
Oct 30 20:41:53.562173 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:41:53.561190    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode936350a_b13c_45e6_ac5a_8d2097b9bba0.slice/crio-539118ad96047fb3b94caf29bc9bc36dda9c77da88e4b808df2036af6b03a7e6 WatchSource:0}: Error finding container 539118ad96047fb3b94caf29bc9bc36dda9c77da88e4b808df2036af6b03a7e6: Status 404 returned error can't find the container with id 539118ad96047fb3b94caf29bc9bc36dda9c77da88e4b808df2036af6b03a7e6
Oct 30 20:41:54.354016 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:54.353976    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e580a5-ftxs5" event={"ID":"e936350a-b13c-45e6-ac5a-8d2097b9bba0","Type":"ContainerStarted","Data":"97b6f26117d793db59de994ec374fa9dd199723e92fc824cde25c66b304a32f3"}
Oct 30 20:41:54.354202 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:54.354044    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e580a5-ftxs5" event={"ID":"e936350a-b13c-45e6-ac5a-8d2097b9bba0","Type":"ContainerStarted","Data":"539118ad96047fb3b94caf29bc9bc36dda9c77da88e4b808df2036af6b03a7e6"}
Oct 30 20:41:54.481596 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:54.481547    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e580a5-ftxs5" podStartSLOduration=2.481532978 podStartE2EDuration="2.481532978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:54.476226572 +0000 UTC m=+926.393248121" watchObservedRunningTime="2026-10-30 20:41:54.481532978 +0000 UTC m=+926.398554541"
Oct 30 20:41:54.732639 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:54.732550    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:41:55.364889 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:55.364851    5071 generic.go:356] "Generic (PLEG): container finished" podID="e936350a-b13c-45e6-ac5a-8d2097b9bba0" containerID="97b6f26117d793db59de994ec374fa9dd199723e92fc824cde25c66b304a32f3" exitCode=0
Oct 30 20:41:55.365060 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:55.364985    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e580a5-ftxs5" event={"ID":"e936350a-b13c-45e6-ac5a-8d2097b9bba0","Type":"ContainerDied","Data":"97b6f26117d793db59de994ec374fa9dd199723e92fc824cde25c66b304a32f3"}
Oct 30 20:41:56.550680 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:56.550656    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e580a5-ftxs5"
Oct 30 20:41:56.671036 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:56.670953    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e936350a-b13c-45e6-ac5a-8d2097b9bba0-kube-api-access-nb2fc\" (UniqueName: \"kubernetes.io/projected/e936350a-b13c-45e6-ac5a-8d2097b9bba0-kube-api-access-nb2fc\") pod \"e936350a-b13c-45e6-ac5a-8d2097b9bba0\" (UID: \"e936350a-b13c-45e6-ac5a-8d2097b9bba0\") "
Oct 30 20:41:56.671036 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:56.671006    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e936350a-b13c-45e6-ac5a-8d2097b9bba0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e936350a-b13c-45e6-ac5a-8d2097b9bba0-maas-api-xks-ca\") pod \"e936350a-b13c-45e6-ac5a-8d2097b9bba0\" (UID: \"e936350a-b13c-45e6-ac5a-8d2097b9bba0\") "
Oct 30 20:41:56.671260 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:56.671121    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e936350a-b13c-45e6-ac5a-8d2097b9bba0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e936350a-b13c-45e6-ac5a-8d2097b9bba0-maas-api-openshift-ca\") pod \"e936350a-b13c-45e6-ac5a-8d2097b9bba0\" (UID: \"e936350a-b13c-45e6-ac5a-8d2097b9bba0\") "
Oct 30 20:41:56.671889 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:56.671812    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e936350a-b13c-45e6-ac5a-8d2097b9bba0-maas-api-openshift-ca" pod "e936350a-b13c-45e6-ac5a-8d2097b9bba0" (UID: "e936350a-b13c-45e6-ac5a-8d2097b9bba0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:41:56.674150 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:56.674116    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e936350a-b13c-45e6-ac5a-8d2097b9bba0-kube-api-access-nb2fc" pod "e936350a-b13c-45e6-ac5a-8d2097b9bba0" (UID: "e936350a-b13c-45e6-ac5a-8d2097b9bba0"). InnerVolumeSpecName "kube-api-access-nb2fc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:41:56.674150 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:56.674143    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e936350a-b13c-45e6-ac5a-8d2097b9bba0-maas-api-xks-ca" pod "e936350a-b13c-45e6-ac5a-8d2097b9bba0" (UID: "e936350a-b13c-45e6-ac5a-8d2097b9bba0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:41:56.773092 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:56.773056    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e936350a-b13c-45e6-ac5a-8d2097b9bba0-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:56.773092 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:56.773086    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e936350a-b13c-45e6-ac5a-8d2097b9bba0-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:56.773293 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:56.773102    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb2fc\" (UniqueName: \"kubernetes.io/projected/e936350a-b13c-45e6-ac5a-8d2097b9bba0-kube-api-access-nb2fc\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:57.378731 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:57.378697    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e580a5-ftxs5"
Oct 30 20:41:57.378918 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:57.378724    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e580a5-ftxs5" event={"ID":"e936350a-b13c-45e6-ac5a-8d2097b9bba0","Type":"ContainerDied","Data":"539118ad96047fb3b94caf29bc9bc36dda9c77da88e4b808df2036af6b03a7e6"}
Oct 30 20:41:57.378918 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:57.378785    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="539118ad96047fb3b94caf29bc9bc36dda9c77da88e4b808df2036af6b03a7e6"
Oct 30 20:41:58.346921 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:58.346880    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3041002-jsp28"]
Oct 30 20:41:58.347613 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:58.347591    5071 cpu_manager.go:414] "containerMap: removing container" podUID="e936350a-b13c-45e6-ac5a-8d2097b9bba0" containerName="revoke-keys"
Oct 30 20:41:58.347721 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:58.347615    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e936350a-b13c-45e6-ac5a-8d2097b9bba0" containerName="revoke-keys"
Oct 30 20:41:58.347780 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:58.347768    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="e936350a-b13c-45e6-ac5a-8d2097b9bba0" containerName="revoke-keys"
Oct 30 20:41:58.351955 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:58.351932    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3041002-jsp28"
Oct 30 20:41:58.387105 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:58.387075    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs5dc\" (UniqueName: \"kubernetes.io/projected/36b847ec-944b-442b-a43d-7334f32eba81-kube-api-access-vs5dc\") pod \"maas-api-revoke-sub-e3041002-jsp28\" (UID: \"36b847ec-944b-442b-a43d-7334f32eba81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3041002-jsp28"
Oct 30 20:41:58.387247 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:58.387111    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36b847ec-944b-442b-a43d-7334f32eba81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3041002-jsp28\" (UID: \"36b847ec-944b-442b-a43d-7334f32eba81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3041002-jsp28"
Oct 30 20:41:58.387247 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:58.387133    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36b847ec-944b-442b-a43d-7334f32eba81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3041002-jsp28\" (UID: \"36b847ec-944b-442b-a43d-7334f32eba81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3041002-jsp28"
Oct 30 20:41:58.458293 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:58.458265    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:41:58.458293 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:58.458285    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 20:41:58.461234 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:58.461213    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3041002-jsp28"]
Oct 30 20:41:58.488044 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:58.488016    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs5dc\" (UniqueName: \"kubernetes.io/projected/36b847ec-944b-442b-a43d-7334f32eba81-kube-api-access-vs5dc\") pod \"maas-api-revoke-sub-e3041002-jsp28\" (UID: \"36b847ec-944b-442b-a43d-7334f32eba81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3041002-jsp28"
Oct 30 20:41:58.488044 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:58.488046    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36b847ec-944b-442b-a43d-7334f32eba81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3041002-jsp28\" (UID: \"36b847ec-944b-442b-a43d-7334f32eba81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3041002-jsp28"
Oct 30 20:41:58.488202 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:58.488069    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36b847ec-944b-442b-a43d-7334f32eba81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3041002-jsp28\" (UID: \"36b847ec-944b-442b-a43d-7334f32eba81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3041002-jsp28"
Oct 30 20:41:58.488623 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:58.488606    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36b847ec-944b-442b-a43d-7334f32eba81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3041002-jsp28\" (UID: \"36b847ec-944b-442b-a43d-7334f32eba81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3041002-jsp28"
Oct 30 20:41:58.490672 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:58.490642    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36b847ec-944b-442b-a43d-7334f32eba81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3041002-jsp28\" (UID: \"36b847ec-944b-442b-a43d-7334f32eba81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3041002-jsp28"
Oct 30 20:41:58.552866 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:58.552808    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs5dc\" (UniqueName: \"kubernetes.io/projected/36b847ec-944b-442b-a43d-7334f32eba81-kube-api-access-vs5dc\") pod \"maas-api-revoke-sub-e3041002-jsp28\" (UID: \"36b847ec-944b-442b-a43d-7334f32eba81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3041002-jsp28"
Oct 30 20:41:58.666215 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:58.666118    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3041002-jsp28"
Oct 30 20:41:59.043646 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:59.043614    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3041002-jsp28"]
Oct 30 20:41:59.043895 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:41:59.043807    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36b847ec_944b_442b_a43d_7334f32eba81.slice/crio-9d3d3130a86f8d2ffd1abfbec340e71bc804494f2cc6542e87bf26c37e5149f8 WatchSource:0}: Error finding container 9d3d3130a86f8d2ffd1abfbec340e71bc804494f2cc6542e87bf26c37e5149f8: Status 404 returned error can't find the container with id 9d3d3130a86f8d2ffd1abfbec340e71bc804494f2cc6542e87bf26c37e5149f8
Oct 30 20:41:59.393464 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:59.393426    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3041002-jsp28" event={"ID":"36b847ec-944b-442b-a43d-7334f32eba81","Type":"ContainerStarted","Data":"e25e5bafb126a74b31fa521055153a0aa6e480158a57ef92d2a55f0907b8ec28"}
Oct 30 20:41:59.393896 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:59.393476    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3041002-jsp28" event={"ID":"36b847ec-944b-442b-a43d-7334f32eba81","Type":"ContainerStarted","Data":"9d3d3130a86f8d2ffd1abfbec340e71bc804494f2cc6542e87bf26c37e5149f8"}
Oct 30 20:41:59.510119 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:59.510069    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3041002-jsp28" podStartSLOduration=1.510055664 podStartE2EDuration="1.510055664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:59.510000646 +0000 UTC m=+931.427022185" watchObservedRunningTime="2026-10-30 20:41:59.510055664 +0000 UTC m=+931.427077200"
Oct 30 20:41:59.700210 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:41:59.700113    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:00.402199 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:00.402165    5071 generic.go:356] "Generic (PLEG): container finished" podID="36b847ec-944b-442b-a43d-7334f32eba81" containerID="e25e5bafb126a74b31fa521055153a0aa6e480158a57ef92d2a55f0907b8ec28" exitCode=0
Oct 30 20:42:00.402597 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:00.402216    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3041002-jsp28" event={"ID":"36b847ec-944b-442b-a43d-7334f32eba81","Type":"ContainerDied","Data":"e25e5bafb126a74b31fa521055153a0aa6e480158a57ef92d2a55f0907b8ec28"}
Oct 30 20:42:01.572316 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:01.572289    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3041002-jsp28"
Oct 30 20:42:01.622572 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:01.622534    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36b847ec-944b-442b-a43d-7334f32eba81-kube-api-access-vs5dc\" (UniqueName: \"kubernetes.io/projected/36b847ec-944b-442b-a43d-7334f32eba81-kube-api-access-vs5dc\") pod \"36b847ec-944b-442b-a43d-7334f32eba81\" (UID: \"36b847ec-944b-442b-a43d-7334f32eba81\") "
Oct 30 20:42:01.622765 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:01.622593    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36b847ec-944b-442b-a43d-7334f32eba81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36b847ec-944b-442b-a43d-7334f32eba81-maas-api-openshift-ca\") pod \"36b847ec-944b-442b-a43d-7334f32eba81\" (UID: \"36b847ec-944b-442b-a43d-7334f32eba81\") "
Oct 30 20:42:01.622765 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:01.622750    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36b847ec-944b-442b-a43d-7334f32eba81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36b847ec-944b-442b-a43d-7334f32eba81-maas-api-xks-ca\") pod \"36b847ec-944b-442b-a43d-7334f32eba81\" (UID: \"36b847ec-944b-442b-a43d-7334f32eba81\") "
Oct 30 20:42:01.623024 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:01.622994    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36b847ec-944b-442b-a43d-7334f32eba81-maas-api-openshift-ca" pod "36b847ec-944b-442b-a43d-7334f32eba81" (UID: "36b847ec-944b-442b-a43d-7334f32eba81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:42:01.623113 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:01.623101    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36b847ec-944b-442b-a43d-7334f32eba81-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:01.625121 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:01.625094    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b847ec-944b-442b-a43d-7334f32eba81-kube-api-access-vs5dc" pod "36b847ec-944b-442b-a43d-7334f32eba81" (UID: "36b847ec-944b-442b-a43d-7334f32eba81"). InnerVolumeSpecName "kube-api-access-vs5dc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:42:01.625121 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:01.625097    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36b847ec-944b-442b-a43d-7334f32eba81-maas-api-xks-ca" pod "36b847ec-944b-442b-a43d-7334f32eba81" (UID: "36b847ec-944b-442b-a43d-7334f32eba81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:42:01.724113 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:01.724026    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs5dc\" (UniqueName: \"kubernetes.io/projected/36b847ec-944b-442b-a43d-7334f32eba81-kube-api-access-vs5dc\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:01.724113 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:01.724063    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36b847ec-944b-442b-a43d-7334f32eba81-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:02.416806 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:02.416772    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3041002-jsp28"
Oct 30 20:42:02.416994 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:02.416805    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3041002-jsp28" event={"ID":"36b847ec-944b-442b-a43d-7334f32eba81","Type":"ContainerDied","Data":"9d3d3130a86f8d2ffd1abfbec340e71bc804494f2cc6542e87bf26c37e5149f8"}
Oct 30 20:42:02.416994 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:02.416887    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d3d3130a86f8d2ffd1abfbec340e71bc804494f2cc6542e87bf26c37e5149f8"
Oct 30 20:42:04.697629 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:04.697595    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:05.142951 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:05.142916    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:09.291547 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:09.291469    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1313706-889b7"]
Oct 30 20:42:09.292053 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:09.292037    5071 cpu_manager.go:414] "containerMap: removing container" podUID="36b847ec-944b-442b-a43d-7334f32eba81" containerName="revoke-keys"
Oct 30 20:42:09.292096 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:09.292054    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36b847ec-944b-442b-a43d-7334f32eba81" containerName="revoke-keys"
Oct 30 20:42:09.292175 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:09.292166    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="36b847ec-944b-442b-a43d-7334f32eba81" containerName="revoke-keys"
Oct 30 20:42:09.295930 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:09.295907    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1313706-889b7"
Oct 30 20:42:09.354412 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:09.354381    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1313706-889b7"]
Oct 30 20:42:09.399608 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:09.399583    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:42:09.399777 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:09.399634    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 20:42:09.409666 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:09.409640    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c977357-4ddf-4cfc-925b-7019c62edee8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1313706-889b7\" (UID: \"2c977357-4ddf-4cfc-925b-7019c62edee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1313706-889b7"
Oct 30 20:42:09.409810 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:09.409754    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhv2g\" (UniqueName: \"kubernetes.io/projected/2c977357-4ddf-4cfc-925b-7019c62edee8-kube-api-access-hhv2g\") pod \"maas-api-revoke-sub-c1313706-889b7\" (UID: \"2c977357-4ddf-4cfc-925b-7019c62edee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1313706-889b7"
Oct 30 20:42:09.409930 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:09.409911    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c977357-4ddf-4cfc-925b-7019c62edee8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1313706-889b7\" (UID: \"2c977357-4ddf-4cfc-925b-7019c62edee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1313706-889b7"
Oct 30 20:42:09.511338 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:09.511307    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhv2g\" (UniqueName: \"kubernetes.io/projected/2c977357-4ddf-4cfc-925b-7019c62edee8-kube-api-access-hhv2g\") pod \"maas-api-revoke-sub-c1313706-889b7\" (UID: \"2c977357-4ddf-4cfc-925b-7019c62edee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1313706-889b7"
Oct 30 20:42:09.511507 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:09.511399    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c977357-4ddf-4cfc-925b-7019c62edee8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1313706-889b7\" (UID: \"2c977357-4ddf-4cfc-925b-7019c62edee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1313706-889b7"
Oct 30 20:42:09.511507 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:09.511467    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c977357-4ddf-4cfc-925b-7019c62edee8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1313706-889b7\" (UID: \"2c977357-4ddf-4cfc-925b-7019c62edee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1313706-889b7"
Oct 30 20:42:09.512083 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:09.512061    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c977357-4ddf-4cfc-925b-7019c62edee8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1313706-889b7\" (UID: \"2c977357-4ddf-4cfc-925b-7019c62edee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1313706-889b7"
Oct 30 20:42:09.514013 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:09.513989    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c977357-4ddf-4cfc-925b-7019c62edee8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1313706-889b7\" (UID: \"2c977357-4ddf-4cfc-925b-7019c62edee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1313706-889b7"
Oct 30 20:42:09.531106 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:09.531082    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6a5c3ab-k6cfs"]
Oct 30 20:42:09.535097 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:09.535081    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6a5c3ab-k6cfs"
Oct 30 20:42:09.570353 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:09.570327    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhv2g\" (UniqueName: \"kubernetes.io/projected/2c977357-4ddf-4cfc-925b-7019c62edee8-kube-api-access-hhv2g\") pod \"maas-api-revoke-sub-c1313706-889b7\" (UID: \"2c977357-4ddf-4cfc-925b-7019c62edee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1313706-889b7"
Oct 30 20:42:09.596702 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:09.596669    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6a5c3ab-k6cfs"]
Oct 30 20:42:09.609138 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:09.609105    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1313706-889b7"
Oct 30 20:42:09.613252 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:09.612523    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsbb5\" (UniqueName: \"kubernetes.io/projected/01b55b11-863d-4088-a049-29480079f206-kube-api-access-tsbb5\") pod \"maas-api-revoke-sub-d6a5c3ab-k6cfs\" (UID: \"01b55b11-863d-4088-a049-29480079f206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6a5c3ab-k6cfs"
Oct 30 20:42:09.613252 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:09.612588    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01b55b11-863d-4088-a049-29480079f206-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6a5c3ab-k6cfs\" (UID: \"01b55b11-863d-4088-a049-29480079f206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6a5c3ab-k6cfs"
Oct 30 20:42:09.613252 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:09.612630    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01b55b11-863d-4088-a049-29480079f206-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6a5c3ab-k6cfs\" (UID: \"01b55b11-863d-4088-a049-29480079f206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6a5c3ab-k6cfs"
Oct 30 20:42:09.689037 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:09.688973    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:09.723078 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:09.721886    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsbb5\" (UniqueName: \"kubernetes.io/projected/01b55b11-863d-4088-a049-29480079f206-kube-api-access-tsbb5\") pod \"maas-api-revoke-sub-d6a5c3ab-k6cfs\" (UID: \"01b55b11-863d-4088-a049-29480079f206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6a5c3ab-k6cfs"
Oct 30 20:42:09.723769 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:09.721969    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01b55b11-863d-4088-a049-29480079f206-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6a5c3ab-k6cfs\" (UID: \"01b55b11-863d-4088-a049-29480079f206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6a5c3ab-k6cfs"
Oct 30 20:42:09.723973 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:09.723855    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01b55b11-863d-4088-a049-29480079f206-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6a5c3ab-k6cfs\" (UID: \"01b55b11-863d-4088-a049-29480079f206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6a5c3ab-k6cfs"
Oct 30 20:42:09.724854 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:09.724781    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01b55b11-863d-4088-a049-29480079f206-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6a5c3ab-k6cfs\" (UID: \"01b55b11-863d-4088-a049-29480079f206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6a5c3ab-k6cfs"
Oct 30 20:42:09.746630 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:09.746569    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01b55b11-863d-4088-a049-29480079f206-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6a5c3ab-k6cfs\" (UID: \"01b55b11-863d-4088-a049-29480079f206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6a5c3ab-k6cfs"
Oct 30 20:42:09.786978 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:09.786949    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsbb5\" (UniqueName: \"kubernetes.io/projected/01b55b11-863d-4088-a049-29480079f206-kube-api-access-tsbb5\") pod \"maas-api-revoke-sub-d6a5c3ab-k6cfs\" (UID: \"01b55b11-863d-4088-a049-29480079f206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6a5c3ab-k6cfs"
Oct 30 20:42:09.848944 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:09.848872    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6a5c3ab-k6cfs"
Oct 30 20:42:10.016392 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:10.016359    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1313706-889b7"]
Oct 30 20:42:10.017569 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:42:10.017542    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c977357_4ddf_4cfc_925b_7019c62edee8.slice/crio-0b3d68fa62cabba867d7e5fa961cda76151f384cd391cf83666dd27e5685a96b WatchSource:0}: Error finding container 0b3d68fa62cabba867d7e5fa961cda76151f384cd391cf83666dd27e5685a96b: Status 404 returned error can't find the container with id 0b3d68fa62cabba867d7e5fa961cda76151f384cd391cf83666dd27e5685a96b
Oct 30 20:42:10.231006 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:10.230980    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6a5c3ab-k6cfs"]
Oct 30 20:42:10.232654 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:42:10.232623    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01b55b11_863d_4088_a049_29480079f206.slice/crio-1f99d4fbe1c9f00a4ac5d58f6a4abb9ecb5223a3bde3e007d63448c0730b73ce WatchSource:0}: Error finding container 1f99d4fbe1c9f00a4ac5d58f6a4abb9ecb5223a3bde3e007d63448c0730b73ce: Status 404 returned error can't find the container with id 1f99d4fbe1c9f00a4ac5d58f6a4abb9ecb5223a3bde3e007d63448c0730b73ce
Oct 30 20:42:10.467977 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:10.467939    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6a5c3ab-k6cfs" event={"ID":"01b55b11-863d-4088-a049-29480079f206","Type":"ContainerStarted","Data":"1f99d4fbe1c9f00a4ac5d58f6a4abb9ecb5223a3bde3e007d63448c0730b73ce"}
Oct 30 20:42:10.469681 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:10.469628    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1313706-889b7" event={"ID":"2c977357-4ddf-4cfc-925b-7019c62edee8","Type":"ContainerStarted","Data":"f111f62c56b575b4e35eb61fa0903f36968c214fb7b213a528e692ecc0c5e648"}
Oct 30 20:42:10.469866 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:10.469700    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1313706-889b7" event={"ID":"2c977357-4ddf-4cfc-925b-7019c62edee8","Type":"ContainerStarted","Data":"0b3d68fa62cabba867d7e5fa961cda76151f384cd391cf83666dd27e5685a96b"}
Oct 30 20:42:10.587106 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:10.587053    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1313706-889b7" podStartSLOduration=1.587037158 podStartE2EDuration="1.587037158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:10.584712291 +0000 UTC m=+942.501733828" watchObservedRunningTime="2026-10-30 20:42:10.587037158 +0000 UTC m=+942.504058694"
Oct 30 20:42:11.477642 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:11.477551    5071 generic.go:356] "Generic (PLEG): container finished" podID="01b55b11-863d-4088-a049-29480079f206" containerID="f2a0f968594a25a1e594bbcc3a067ad10b1d7bc5069a685bd63ff1cddc8ae17a" exitCode=0
Oct 30 20:42:11.478084 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:11.477639    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6a5c3ab-k6cfs" event={"ID":"01b55b11-863d-4088-a049-29480079f206","Type":"ContainerDied","Data":"f2a0f968594a25a1e594bbcc3a067ad10b1d7bc5069a685bd63ff1cddc8ae17a"}
Oct 30 20:42:11.479327 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:11.479302    5071 generic.go:356] "Generic (PLEG): container finished" podID="2c977357-4ddf-4cfc-925b-7019c62edee8" containerID="f111f62c56b575b4e35eb61fa0903f36968c214fb7b213a528e692ecc0c5e648" exitCode=0
Oct 30 20:42:11.479462 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:11.479368    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1313706-889b7" event={"ID":"2c977357-4ddf-4cfc-925b-7019c62edee8","Type":"ContainerDied","Data":"f111f62c56b575b4e35eb61fa0903f36968c214fb7b213a528e692ecc0c5e648"}
Oct 30 20:42:12.655883 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:12.655856    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6a5c3ab-k6cfs"
Oct 30 20:42:12.680328 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:12.680303    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1313706-889b7"
Oct 30 20:42:12.757301 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:12.757210    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01b55b11-863d-4088-a049-29480079f206-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01b55b11-863d-4088-a049-29480079f206-maas-api-openshift-ca\") pod \"01b55b11-863d-4088-a049-29480079f206\" (UID: \"01b55b11-863d-4088-a049-29480079f206\") "
Oct 30 20:42:12.757301 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:12.757270    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01b55b11-863d-4088-a049-29480079f206-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01b55b11-863d-4088-a049-29480079f206-maas-api-xks-ca\") pod \"01b55b11-863d-4088-a049-29480079f206\" (UID: \"01b55b11-863d-4088-a049-29480079f206\") "
Oct 30 20:42:12.757508 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:12.757415    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01b55b11-863d-4088-a049-29480079f206-kube-api-access-tsbb5\" (UniqueName: \"kubernetes.io/projected/01b55b11-863d-4088-a049-29480079f206-kube-api-access-tsbb5\") pod \"01b55b11-863d-4088-a049-29480079f206\" (UID: \"01b55b11-863d-4088-a049-29480079f206\") "
Oct 30 20:42:12.757653 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:12.757631    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b55b11-863d-4088-a049-29480079f206-maas-api-openshift-ca" pod "01b55b11-863d-4088-a049-29480079f206" (UID: "01b55b11-863d-4088-a049-29480079f206"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:42:12.758035 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:12.757966    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01b55b11-863d-4088-a049-29480079f206-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:12.759799 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:12.759771    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01b55b11-863d-4088-a049-29480079f206-kube-api-access-tsbb5" pod "01b55b11-863d-4088-a049-29480079f206" (UID: "01b55b11-863d-4088-a049-29480079f206"). InnerVolumeSpecName "kube-api-access-tsbb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:42:12.759966 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:12.759943    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01b55b11-863d-4088-a049-29480079f206-maas-api-xks-ca" pod "01b55b11-863d-4088-a049-29480079f206" (UID: "01b55b11-863d-4088-a049-29480079f206"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:42:12.858966 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:12.858933    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c977357-4ddf-4cfc-925b-7019c62edee8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c977357-4ddf-4cfc-925b-7019c62edee8-maas-api-xks-ca\") pod \"2c977357-4ddf-4cfc-925b-7019c62edee8\" (UID: \"2c977357-4ddf-4cfc-925b-7019c62edee8\") "
Oct 30 20:42:12.859163 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:12.859008    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c977357-4ddf-4cfc-925b-7019c62edee8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c977357-4ddf-4cfc-925b-7019c62edee8-maas-api-openshift-ca\") pod \"2c977357-4ddf-4cfc-925b-7019c62edee8\" (UID: \"2c977357-4ddf-4cfc-925b-7019c62edee8\") "
Oct 30 20:42:12.859243 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:12.859203    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c977357-4ddf-4cfc-925b-7019c62edee8-kube-api-access-hhv2g\" (UniqueName: \"kubernetes.io/projected/2c977357-4ddf-4cfc-925b-7019c62edee8-kube-api-access-hhv2g\") pod \"2c977357-4ddf-4cfc-925b-7019c62edee8\" (UID: \"2c977357-4ddf-4cfc-925b-7019c62edee8\") "
Oct 30 20:42:12.859522 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:12.859495    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c977357-4ddf-4cfc-925b-7019c62edee8-maas-api-openshift-ca" pod "2c977357-4ddf-4cfc-925b-7019c62edee8" (UID: "2c977357-4ddf-4cfc-925b-7019c62edee8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:42:12.859654 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:12.859602    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01b55b11-863d-4088-a049-29480079f206-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:12.859654 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:12.859621    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c977357-4ddf-4cfc-925b-7019c62edee8-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:12.859654 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:12.859637    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsbb5\" (UniqueName: \"kubernetes.io/projected/01b55b11-863d-4088-a049-29480079f206-kube-api-access-tsbb5\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:12.862174 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:12.862146    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c977357-4ddf-4cfc-925b-7019c62edee8-maas-api-xks-ca" pod "2c977357-4ddf-4cfc-925b-7019c62edee8" (UID: "2c977357-4ddf-4cfc-925b-7019c62edee8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:42:12.862306 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:12.862289    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c977357-4ddf-4cfc-925b-7019c62edee8-kube-api-access-hhv2g" pod "2c977357-4ddf-4cfc-925b-7019c62edee8" (UID: "2c977357-4ddf-4cfc-925b-7019c62edee8"). InnerVolumeSpecName "kube-api-access-hhv2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:42:12.961249 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:12.961216    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhv2g\" (UniqueName: \"kubernetes.io/projected/2c977357-4ddf-4cfc-925b-7019c62edee8-kube-api-access-hhv2g\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:12.961249 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:12.961247    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c977357-4ddf-4cfc-925b-7019c62edee8-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:13.492753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:13.492724    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1313706-889b7"
Oct 30 20:42:13.492959 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:13.492763    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1313706-889b7" event={"ID":"2c977357-4ddf-4cfc-925b-7019c62edee8","Type":"ContainerDied","Data":"0b3d68fa62cabba867d7e5fa961cda76151f384cd391cf83666dd27e5685a96b"}
Oct 30 20:42:13.492959 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:13.492811    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b3d68fa62cabba867d7e5fa961cda76151f384cd391cf83666dd27e5685a96b"
Oct 30 20:42:13.494106 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:13.494070    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6a5c3ab-k6cfs" event={"ID":"01b55b11-863d-4088-a049-29480079f206","Type":"ContainerDied","Data":"1f99d4fbe1c9f00a4ac5d58f6a4abb9ecb5223a3bde3e007d63448c0730b73ce"}
Oct 30 20:42:13.494106 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:13.494088    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6a5c3ab-k6cfs"
Oct 30 20:42:13.494269 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:13.494116    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f99d4fbe1c9f00a4ac5d58f6a4abb9ecb5223a3bde3e007d63448c0730b73ce"
Oct 30 20:42:14.055881 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:14.055845    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f4d9d61-5bqhp"]
Oct 30 20:42:14.056410 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:14.056397    5071 cpu_manager.go:414] "containerMap: removing container" podUID="01b55b11-863d-4088-a049-29480079f206" containerName="revoke-keys"
Oct 30 20:42:14.056477 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:14.056412    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01b55b11-863d-4088-a049-29480079f206" containerName="revoke-keys"
Oct 30 20:42:14.056477 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:14.056438    5071 cpu_manager.go:414] "containerMap: removing container" podUID="2c977357-4ddf-4cfc-925b-7019c62edee8" containerName="revoke-keys"
Oct 30 20:42:14.056477 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:14.056443    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c977357-4ddf-4cfc-925b-7019c62edee8" containerName="revoke-keys"
Oct 30 20:42:14.056639 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:14.056510    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c977357-4ddf-4cfc-925b-7019c62edee8" containerName="revoke-keys"
Oct 30 20:42:14.056639 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:14.056522    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="01b55b11-863d-4088-a049-29480079f206" containerName="revoke-keys"
Oct 30 20:42:14.060150 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:14.060130    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f4d9d61-5bqhp"
Oct 30 20:42:14.120852 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:14.120805    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f4d9d61-5bqhp"]
Oct 30 20:42:14.165973 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:14.165944    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 20:42:14.165973 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:14.165957    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:42:14.174808 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:14.174779    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lnfm\" (UniqueName: \"kubernetes.io/projected/5c1c144f-aa0b-4d88-a3a5-4c399288fd1c-kube-api-access-8lnfm\") pod \"maas-api-revoke-sub-7f4d9d61-5bqhp\" (UID: \"5c1c144f-aa0b-4d88-a3a5-4c399288fd1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f4d9d61-5bqhp"
Oct 30 20:42:14.174930 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:14.174858    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c1c144f-aa0b-4d88-a3a5-4c399288fd1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f4d9d61-5bqhp\" (UID: \"5c1c144f-aa0b-4d88-a3a5-4c399288fd1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f4d9d61-5bqhp"
Oct 30 20:42:14.175007 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:14.174988    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c1c144f-aa0b-4d88-a3a5-4c399288fd1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f4d9d61-5bqhp\" (UID: \"5c1c144f-aa0b-4d88-a3a5-4c399288fd1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f4d9d61-5bqhp"
Oct 30 20:42:14.276333 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:14.276305    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c1c144f-aa0b-4d88-a3a5-4c399288fd1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f4d9d61-5bqhp\" (UID: \"5c1c144f-aa0b-4d88-a3a5-4c399288fd1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f4d9d61-5bqhp"
Oct 30 20:42:14.276505 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:14.276422    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lnfm\" (UniqueName: \"kubernetes.io/projected/5c1c144f-aa0b-4d88-a3a5-4c399288fd1c-kube-api-access-8lnfm\") pod \"maas-api-revoke-sub-7f4d9d61-5bqhp\" (UID: \"5c1c144f-aa0b-4d88-a3a5-4c399288fd1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f4d9d61-5bqhp"
Oct 30 20:42:14.276505 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:14.276448    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c1c144f-aa0b-4d88-a3a5-4c399288fd1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f4d9d61-5bqhp\" (UID: \"5c1c144f-aa0b-4d88-a3a5-4c399288fd1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f4d9d61-5bqhp"
Oct 30 20:42:14.277174 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:14.277145    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c1c144f-aa0b-4d88-a3a5-4c399288fd1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f4d9d61-5bqhp\" (UID: \"5c1c144f-aa0b-4d88-a3a5-4c399288fd1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f4d9d61-5bqhp"
Oct 30 20:42:14.278942 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:14.278918    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c1c144f-aa0b-4d88-a3a5-4c399288fd1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f4d9d61-5bqhp\" (UID: \"5c1c144f-aa0b-4d88-a3a5-4c399288fd1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f4d9d61-5bqhp"
Oct 30 20:42:14.336030 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:14.335956    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lnfm\" (UniqueName: \"kubernetes.io/projected/5c1c144f-aa0b-4d88-a3a5-4c399288fd1c-kube-api-access-8lnfm\") pod \"maas-api-revoke-sub-7f4d9d61-5bqhp\" (UID: \"5c1c144f-aa0b-4d88-a3a5-4c399288fd1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f4d9d61-5bqhp"
Oct 30 20:42:14.374637 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:14.374607    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f4d9d61-5bqhp"
Oct 30 20:42:14.767739 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:14.767712    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f4d9d61-5bqhp"]
Oct 30 20:42:14.769983 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:42:14.769954    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c1c144f_aa0b_4d88_a3a5_4c399288fd1c.slice/crio-08b175c6ede58a162a591bbc807163873ce01c2cdea7dc7a75c6ef0c4a63c590 WatchSource:0}: Error finding container 08b175c6ede58a162a591bbc807163873ce01c2cdea7dc7a75c6ef0c4a63c590: Status 404 returned error can't find the container with id 08b175c6ede58a162a591bbc807163873ce01c2cdea7dc7a75c6ef0c4a63c590
Oct 30 20:42:14.876765 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:14.876730    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:15.509669 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:15.509579    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f4d9d61-5bqhp" event={"ID":"5c1c144f-aa0b-4d88-a3a5-4c399288fd1c","Type":"ContainerStarted","Data":"7b62535ac9ae1725ab2e66b73f5f33456f6389d8294a0f850dd96822162fcfa6"}
Oct 30 20:42:15.509669 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:15.509659    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f4d9d61-5bqhp" event={"ID":"5c1c144f-aa0b-4d88-a3a5-4c399288fd1c","Type":"ContainerStarted","Data":"08b175c6ede58a162a591bbc807163873ce01c2cdea7dc7a75c6ef0c4a63c590"}
Oct 30 20:42:15.629171 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:15.629126    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f4d9d61-5bqhp" podStartSLOduration=1.629111714 podStartE2EDuration="1.629111714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:15.626214563 +0000 UTC m=+947.543236100" watchObservedRunningTime="2026-10-30 20:42:15.629111714 +0000 UTC m=+947.546133251"
Oct 30 20:42:16.021142 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:16.021099    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:42:16.518401 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:16.518368    5071 generic.go:356] "Generic (PLEG): container finished" podID="5c1c144f-aa0b-4d88-a3a5-4c399288fd1c" containerID="7b62535ac9ae1725ab2e66b73f5f33456f6389d8294a0f850dd96822162fcfa6" exitCode=0
Oct 30 20:42:16.518810 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:16.518450    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f4d9d61-5bqhp" event={"ID":"5c1c144f-aa0b-4d88-a3a5-4c399288fd1c","Type":"ContainerDied","Data":"7b62535ac9ae1725ab2e66b73f5f33456f6389d8294a0f850dd96822162fcfa6"}
Oct 30 20:42:16.765584 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:42:16.765547    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:42:17.700463 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:17.700435    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f4d9d61-5bqhp"
Oct 30 20:42:17.815650 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:17.815616    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c1c144f-aa0b-4d88-a3a5-4c399288fd1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c1c144f-aa0b-4d88-a3a5-4c399288fd1c-maas-api-openshift-ca\") pod \"5c1c144f-aa0b-4d88-a3a5-4c399288fd1c\" (UID: \"5c1c144f-aa0b-4d88-a3a5-4c399288fd1c\") "
Oct 30 20:42:17.815906 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:17.815691    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c1c144f-aa0b-4d88-a3a5-4c399288fd1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c1c144f-aa0b-4d88-a3a5-4c399288fd1c-maas-api-xks-ca\") pod \"5c1c144f-aa0b-4d88-a3a5-4c399288fd1c\" (UID: \"5c1c144f-aa0b-4d88-a3a5-4c399288fd1c\") "
Oct 30 20:42:17.815906 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:17.815747    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c1c144f-aa0b-4d88-a3a5-4c399288fd1c-kube-api-access-8lnfm\" (UniqueName: \"kubernetes.io/projected/5c1c144f-aa0b-4d88-a3a5-4c399288fd1c-kube-api-access-8lnfm\") pod \"5c1c144f-aa0b-4d88-a3a5-4c399288fd1c\" (UID: \"5c1c144f-aa0b-4d88-a3a5-4c399288fd1c\") "
Oct 30 20:42:17.816505 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:17.816478    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c1c144f-aa0b-4d88-a3a5-4c399288fd1c-maas-api-openshift-ca" pod "5c1c144f-aa0b-4d88-a3a5-4c399288fd1c" (UID: "5c1c144f-aa0b-4d88-a3a5-4c399288fd1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:42:17.819047 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:17.819007    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c1c144f-aa0b-4d88-a3a5-4c399288fd1c-maas-api-xks-ca" pod "5c1c144f-aa0b-4d88-a3a5-4c399288fd1c" (UID: "5c1c144f-aa0b-4d88-a3a5-4c399288fd1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:42:17.819047 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:17.819027    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c1c144f-aa0b-4d88-a3a5-4c399288fd1c-kube-api-access-8lnfm" pod "5c1c144f-aa0b-4d88-a3a5-4c399288fd1c" (UID: "5c1c144f-aa0b-4d88-a3a5-4c399288fd1c"). InnerVolumeSpecName "kube-api-access-8lnfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:42:17.917226 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:17.917192    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c1c144f-aa0b-4d88-a3a5-4c399288fd1c-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:17.917226 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:17.917220    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lnfm\" (UniqueName: \"kubernetes.io/projected/5c1c144f-aa0b-4d88-a3a5-4c399288fd1c-kube-api-access-8lnfm\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:17.917423 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:17.917235    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c1c144f-aa0b-4d88-a3a5-4c399288fd1c-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:18.532531 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:18.532447    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f4d9d61-5bqhp"
Oct 30 20:42:18.532531 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:18.532455    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f4d9d61-5bqhp" event={"ID":"5c1c144f-aa0b-4d88-a3a5-4c399288fd1c","Type":"ContainerDied","Data":"08b175c6ede58a162a591bbc807163873ce01c2cdea7dc7a75c6ef0c4a63c590"}
Oct 30 20:42:18.532531 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:18.532508    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08b175c6ede58a162a591bbc807163873ce01c2cdea7dc7a75c6ef0c4a63c590"
Oct 30 20:42:19.943604 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:19.941785    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:21.072620 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:21.072586    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:25.076899 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:25.076858    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:25.663976 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:25.663939    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:25.838949 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:25.838913    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:25.858483 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:25.858447    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1318e0ff-jwxgh"]
Oct 30 20:42:25.859320 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:25.859298    5071 cpu_manager.go:414] "containerMap: removing container" podUID="5c1c144f-aa0b-4d88-a3a5-4c399288fd1c" containerName="revoke-keys"
Oct 30 20:42:25.859367 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:25.859325    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c1c144f-aa0b-4d88-a3a5-4c399288fd1c" containerName="revoke-keys"
Oct 30 20:42:25.859507 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:25.859493    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c1c144f-aa0b-4d88-a3a5-4c399288fd1c" containerName="revoke-keys"
Oct 30 20:42:25.864754 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:25.864734    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1318e0ff-jwxgh"
Oct 30 20:42:25.904357 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:25.904326    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4036a76d-bdb3-4542-aae7-50d1bb377050-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1318e0ff-jwxgh\" (UID: \"4036a76d-bdb3-4542-aae7-50d1bb377050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1318e0ff-jwxgh"
Oct 30 20:42:25.904484 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:25.904467    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4036a76d-bdb3-4542-aae7-50d1bb377050-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1318e0ff-jwxgh\" (UID: \"4036a76d-bdb3-4542-aae7-50d1bb377050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1318e0ff-jwxgh"
Oct 30 20:42:25.904574 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:25.904560    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh29f\" (UniqueName: \"kubernetes.io/projected/4036a76d-bdb3-4542-aae7-50d1bb377050-kube-api-access-gh29f\") pod \"maas-api-revoke-sub-1318e0ff-jwxgh\" (UID: \"4036a76d-bdb3-4542-aae7-50d1bb377050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1318e0ff-jwxgh"
Oct 30 20:42:25.921102 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:25.921005    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:42:25.925620 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:25.925597    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1318e0ff-jwxgh"]
Oct 30 20:42:25.970872 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:25.970846    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 20:42:26.005325 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:26.005279    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4036a76d-bdb3-4542-aae7-50d1bb377050-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1318e0ff-jwxgh\" (UID: \"4036a76d-bdb3-4542-aae7-50d1bb377050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1318e0ff-jwxgh"
Oct 30 20:42:26.005481 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:26.005414    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4036a76d-bdb3-4542-aae7-50d1bb377050-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1318e0ff-jwxgh\" (UID: \"4036a76d-bdb3-4542-aae7-50d1bb377050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1318e0ff-jwxgh"
Oct 30 20:42:26.005481 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:26.005466    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh29f\" (UniqueName: \"kubernetes.io/projected/4036a76d-bdb3-4542-aae7-50d1bb377050-kube-api-access-gh29f\") pod \"maas-api-revoke-sub-1318e0ff-jwxgh\" (UID: \"4036a76d-bdb3-4542-aae7-50d1bb377050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1318e0ff-jwxgh"
Oct 30 20:42:26.005942 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:26.005918    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4036a76d-bdb3-4542-aae7-50d1bb377050-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1318e0ff-jwxgh\" (UID: \"4036a76d-bdb3-4542-aae7-50d1bb377050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1318e0ff-jwxgh"
Oct 30 20:42:26.007972 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:26.007952    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4036a76d-bdb3-4542-aae7-50d1bb377050-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1318e0ff-jwxgh\" (UID: \"4036a76d-bdb3-4542-aae7-50d1bb377050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1318e0ff-jwxgh"
Oct 30 20:42:26.065152 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:26.065126    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh29f\" (UniqueName: \"kubernetes.io/projected/4036a76d-bdb3-4542-aae7-50d1bb377050-kube-api-access-gh29f\") pod \"maas-api-revoke-sub-1318e0ff-jwxgh\" (UID: \"4036a76d-bdb3-4542-aae7-50d1bb377050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1318e0ff-jwxgh"
Oct 30 20:42:26.177874 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:26.177753    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1318e0ff-jwxgh"
Oct 30 20:42:26.561332 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:26.561303    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1318e0ff-jwxgh"]
Oct 30 20:42:26.563175 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:42:26.563150    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4036a76d_bdb3_4542_aae7_50d1bb377050.slice/crio-7c4d0e982428933396bce8e9573c1e312ce2d475af8bb23177bf7eafb2a1c9f2 WatchSource:0}: Error finding container 7c4d0e982428933396bce8e9573c1e312ce2d475af8bb23177bf7eafb2a1c9f2: Status 404 returned error can't find the container with id 7c4d0e982428933396bce8e9573c1e312ce2d475af8bb23177bf7eafb2a1c9f2
Oct 30 20:42:26.591203 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:26.591172    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1318e0ff-jwxgh" event={"ID":"4036a76d-bdb3-4542-aae7-50d1bb377050","Type":"ContainerStarted","Data":"7c4d0e982428933396bce8e9573c1e312ce2d475af8bb23177bf7eafb2a1c9f2"}
Oct 30 20:42:26.978020 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:26.977980    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dd79b45-sshj7"]
Oct 30 20:42:26.982228 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:26.982204    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dd79b45-sshj7"
Oct 30 20:42:27.017481 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:27.017454    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r84sc\" (UniqueName: \"kubernetes.io/projected/d7cd5a70-c322-4277-8a96-ebb7b539d088-kube-api-access-r84sc\") pod \"maas-api-revoke-sub-4dd79b45-sshj7\" (UID: \"d7cd5a70-c322-4277-8a96-ebb7b539d088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dd79b45-sshj7"
Oct 30 20:42:27.017628 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:27.017489    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7cd5a70-c322-4277-8a96-ebb7b539d088-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4dd79b45-sshj7\" (UID: \"d7cd5a70-c322-4277-8a96-ebb7b539d088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dd79b45-sshj7"
Oct 30 20:42:27.017628 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:27.017518    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7cd5a70-c322-4277-8a96-ebb7b539d088-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4dd79b45-sshj7\" (UID: \"d7cd5a70-c322-4277-8a96-ebb7b539d088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dd79b45-sshj7"
Oct 30 20:42:27.046054 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:27.046008    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dd79b45-sshj7"]
Oct 30 20:42:27.118362 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:27.118286    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r84sc\" (UniqueName: \"kubernetes.io/projected/d7cd5a70-c322-4277-8a96-ebb7b539d088-kube-api-access-r84sc\") pod \"maas-api-revoke-sub-4dd79b45-sshj7\" (UID: \"d7cd5a70-c322-4277-8a96-ebb7b539d088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dd79b45-sshj7"
Oct 30 20:42:27.118362 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:27.118316    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7cd5a70-c322-4277-8a96-ebb7b539d088-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4dd79b45-sshj7\" (UID: \"d7cd5a70-c322-4277-8a96-ebb7b539d088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dd79b45-sshj7"
Oct 30 20:42:27.118362 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:27.118344    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7cd5a70-c322-4277-8a96-ebb7b539d088-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4dd79b45-sshj7\" (UID: \"d7cd5a70-c322-4277-8a96-ebb7b539d088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dd79b45-sshj7"
Oct 30 20:42:27.119211 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:27.119185    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7cd5a70-c322-4277-8a96-ebb7b539d088-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4dd79b45-sshj7\" (UID: \"d7cd5a70-c322-4277-8a96-ebb7b539d088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dd79b45-sshj7"
Oct 30 20:42:27.120812 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:27.120788    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7cd5a70-c322-4277-8a96-ebb7b539d088-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4dd79b45-sshj7\" (UID: \"d7cd5a70-c322-4277-8a96-ebb7b539d088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dd79b45-sshj7"
Oct 30 20:42:27.177810 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:27.177777    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r84sc\" (UniqueName: \"kubernetes.io/projected/d7cd5a70-c322-4277-8a96-ebb7b539d088-kube-api-access-r84sc\") pod \"maas-api-revoke-sub-4dd79b45-sshj7\" (UID: \"d7cd5a70-c322-4277-8a96-ebb7b539d088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dd79b45-sshj7"
Oct 30 20:42:27.296496 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:27.296465    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dd79b45-sshj7"
Oct 30 20:42:27.598848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:27.598781    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1318e0ff-jwxgh" event={"ID":"4036a76d-bdb3-4542-aae7-50d1bb377050","Type":"ContainerStarted","Data":"4c89174d9b13969cf4efe062cdb0446d2ce3bc1ba2e669ce557415b0e40ed794"}
Oct 30 20:42:27.696974 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:27.696949    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dd79b45-sshj7"]
Oct 30 20:42:27.702046 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:42:27.702015    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7cd5a70_c322_4277_8a96_ebb7b539d088.slice/crio-1257aa73839f7a8135296afd404e4049087eee8677505987d8ca1fe8a64a946b WatchSource:0}: Error finding container 1257aa73839f7a8135296afd404e4049087eee8677505987d8ca1fe8a64a946b: Status 404 returned error can't find the container with id 1257aa73839f7a8135296afd404e4049087eee8677505987d8ca1fe8a64a946b
Oct 30 20:42:27.743155 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:27.743107    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1318e0ff-jwxgh" podStartSLOduration=2.743092881 podStartE2EDuration="2.743092881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:27.742084676 +0000 UTC m=+959.659106211" watchObservedRunningTime="2026-10-30 20:42:27.743092881 +0000 UTC m=+959.660114417"
Oct 30 20:42:27.821688 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:27.821658    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:27.894439 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:27.894363    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"]
Oct 30 20:42:27.899593 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:27.899576    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"
Oct 30 20:42:27.927397 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:27.927370    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05a72fc0-2554-4f73-935e-ade8fb962a7c-istio-data\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g\" (UID: \"05a72fc0-2554-4f73-935e-ade8fb962a7c\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"
Oct 30 20:42:27.927521 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:27.927415    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05a72fc0-2554-4f73-935e-ade8fb962a7c-istio-podinfo\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g\" (UID: \"05a72fc0-2554-4f73-935e-ade8fb962a7c\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"
Oct 30 20:42:27.927521 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:27.927480    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q28mp\" (UniqueName: \"kubernetes.io/projected/05a72fc0-2554-4f73-935e-ade8fb962a7c-kube-api-access-q28mp\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g\" (UID: \"05a72fc0-2554-4f73-935e-ade8fb962a7c\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"
Oct 30 20:42:27.927633 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:27.927537    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05a72fc0-2554-4f73-935e-ade8fb962a7c-istio-envoy\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g\" (UID: \"05a72fc0-2554-4f73-935e-ade8fb962a7c\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"
Oct 30 20:42:27.927699 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:27.927677    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a72fc0-2554-4f73-935e-ade8fb962a7c-credential-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g\" (UID: \"05a72fc0-2554-4f73-935e-ade8fb962a7c\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"
Oct 30 20:42:27.927752 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:27.927733    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05a72fc0-2554-4f73-935e-ade8fb962a7c-istio-token\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g\" (UID: \"05a72fc0-2554-4f73-935e-ade8fb962a7c\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"
Oct 30 20:42:27.927807 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:27.927769    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05a72fc0-2554-4f73-935e-ade8fb962a7c-istiod-ca-cert\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g\" (UID: \"05a72fc0-2554-4f73-935e-ade8fb962a7c\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"
Oct 30 20:42:27.927929 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:27.927913    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a72fc0-2554-4f73-935e-ade8fb962a7c-workload-certs\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g\" (UID: \"05a72fc0-2554-4f73-935e-ade8fb962a7c\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"
Oct 30 20:42:27.927983 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:27.927947    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a72fc0-2554-4f73-935e-ade8fb962a7c-workload-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g\" (UID: \"05a72fc0-2554-4f73-935e-ade8fb962a7c\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"
Oct 30 20:42:27.958464 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:27.958437    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-ee3fad-openshift-default-dockercfg-tzrhl\""
Oct 30 20:42:27.976806 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:27.976778    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"]
Oct 30 20:42:28.028629 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.028603    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q28mp\" (UniqueName: \"kubernetes.io/projected/05a72fc0-2554-4f73-935e-ade8fb962a7c-kube-api-access-q28mp\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g\" (UID: \"05a72fc0-2554-4f73-935e-ade8fb962a7c\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"
Oct 30 20:42:28.028783 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.028669    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05a72fc0-2554-4f73-935e-ade8fb962a7c-istio-envoy\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g\" (UID: \"05a72fc0-2554-4f73-935e-ade8fb962a7c\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"
Oct 30 20:42:28.028928 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.028762    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a72fc0-2554-4f73-935e-ade8fb962a7c-credential-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g\" (UID: \"05a72fc0-2554-4f73-935e-ade8fb962a7c\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"
Oct 30 20:42:28.028928 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.028875    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05a72fc0-2554-4f73-935e-ade8fb962a7c-istio-token\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g\" (UID: \"05a72fc0-2554-4f73-935e-ade8fb962a7c\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"
Oct 30 20:42:28.028928 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.028912    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05a72fc0-2554-4f73-935e-ade8fb962a7c-istiod-ca-cert\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g\" (UID: \"05a72fc0-2554-4f73-935e-ade8fb962a7c\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"
Oct 30 20:42:28.029083 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.028964    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a72fc0-2554-4f73-935e-ade8fb962a7c-workload-certs\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g\" (UID: \"05a72fc0-2554-4f73-935e-ade8fb962a7c\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"
Oct 30 20:42:28.029083 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.028993    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a72fc0-2554-4f73-935e-ade8fb962a7c-workload-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g\" (UID: \"05a72fc0-2554-4f73-935e-ade8fb962a7c\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"
Oct 30 20:42:28.029083 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.029034    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05a72fc0-2554-4f73-935e-ade8fb962a7c-istio-data\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g\" (UID: \"05a72fc0-2554-4f73-935e-ade8fb962a7c\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"
Oct 30 20:42:28.029231 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.029100    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05a72fc0-2554-4f73-935e-ade8fb962a7c-istio-podinfo\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g\" (UID: \"05a72fc0-2554-4f73-935e-ade8fb962a7c\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"
Oct 30 20:42:28.030168 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.030138    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05a72fc0-2554-4f73-935e-ade8fb962a7c-istiod-ca-cert\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g\" (UID: \"05a72fc0-2554-4f73-935e-ade8fb962a7c\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"
Oct 30 20:42:28.030629 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.030606    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a72fc0-2554-4f73-935e-ade8fb962a7c-workload-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g\" (UID: \"05a72fc0-2554-4f73-935e-ade8fb962a7c\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"
Oct 30 20:42:28.030721 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.030606    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a72fc0-2554-4f73-935e-ade8fb962a7c-credential-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g\" (UID: \"05a72fc0-2554-4f73-935e-ade8fb962a7c\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"
Oct 30 20:42:28.030721 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.030699    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05a72fc0-2554-4f73-935e-ade8fb962a7c-istio-data\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g\" (UID: \"05a72fc0-2554-4f73-935e-ade8fb962a7c\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"
Oct 30 20:42:28.030806 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.030726    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a72fc0-2554-4f73-935e-ade8fb962a7c-workload-certs\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g\" (UID: \"05a72fc0-2554-4f73-935e-ade8fb962a7c\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"
Oct 30 20:42:28.031466 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.031445    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05a72fc0-2554-4f73-935e-ade8fb962a7c-istio-envoy\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g\" (UID: \"05a72fc0-2554-4f73-935e-ade8fb962a7c\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"
Oct 30 20:42:28.031969 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.031952    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05a72fc0-2554-4f73-935e-ade8fb962a7c-istio-podinfo\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g\" (UID: \"05a72fc0-2554-4f73-935e-ade8fb962a7c\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"
Oct 30 20:42:28.091447 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.091414    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05a72fc0-2554-4f73-935e-ade8fb962a7c-istio-token\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g\" (UID: \"05a72fc0-2554-4f73-935e-ade8fb962a7c\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"
Oct 30 20:42:28.091788 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.091763    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q28mp\" (UniqueName: \"kubernetes.io/projected/05a72fc0-2554-4f73-935e-ade8fb962a7c-kube-api-access-q28mp\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g\" (UID: \"05a72fc0-2554-4f73-935e-ade8fb962a7c\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"
Oct 30 20:42:28.215356 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.215268    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"]
Oct 30 20:42:28.219223 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.219202    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"
Oct 30 20:42:28.264853 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.264809    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-ee3fad-openshift-default-dockercfg-tzrhl\""
Oct 30 20:42:28.274064 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.274032    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"
Oct 30 20:42:28.277435 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.277411    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-f6ebf5-openshift-default-dockercfg-hsckc\""
Oct 30 20:42:28.286424 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.286396    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"]
Oct 30 20:42:28.333462 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.333419    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-istio-envoy\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87\" (UID: \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"
Oct 30 20:42:28.333783 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.333477    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-istio-podinfo\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87\" (UID: \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"
Oct 30 20:42:28.333783 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.333524    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-istio-data\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87\" (UID: \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"
Oct 30 20:42:28.333783 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.333558    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdxfk\" (UniqueName: \"kubernetes.io/projected/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-kube-api-access-jdxfk\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87\" (UID: \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"
Oct 30 20:42:28.333783 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.333624    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-workload-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87\" (UID: \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"
Oct 30 20:42:28.333783 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.333654    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-istio-token\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87\" (UID: \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"
Oct 30 20:42:28.333783 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.333770    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-workload-certs\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87\" (UID: \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"
Oct 30 20:42:28.334043 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.333797    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-credential-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87\" (UID: \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"
Oct 30 20:42:28.334043 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.333871    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-istiod-ca-cert\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87\" (UID: \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"
Oct 30 20:42:28.434609 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.434575    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdxfk\" (UniqueName: \"kubernetes.io/projected/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-kube-api-access-jdxfk\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87\" (UID: \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"
Oct 30 20:42:28.434801 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.434654    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-workload-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87\" (UID: \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"
Oct 30 20:42:28.434801 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.434682    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-istio-token\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87\" (UID: \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"
Oct 30 20:42:28.434801 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.434784    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-workload-certs\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87\" (UID: \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"
Oct 30 20:42:28.435040 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.434810    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-credential-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87\" (UID: \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"
Oct 30 20:42:28.435040 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.434872    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-istiod-ca-cert\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87\" (UID: \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"
Oct 30 20:42:28.435040 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.434948    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-istio-envoy\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87\" (UID: \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"
Oct 30 20:42:28.435040 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.434976    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-istio-podinfo\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87\" (UID: \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"
Oct 30 20:42:28.435040 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.435021    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-istio-data\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87\" (UID: \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"
Oct 30 20:42:28.435288 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.435172    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-workload-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87\" (UID: \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"
Oct 30 20:42:28.435288 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.435223    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-credential-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87\" (UID: \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"
Oct 30 20:42:28.435389 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.435360    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-workload-certs\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87\" (UID: \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"
Oct 30 20:42:28.435455 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.435388    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-istio-data\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87\" (UID: \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"
Oct 30 20:42:28.436002 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.435947    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-istiod-ca-cert\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87\" (UID: \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"
Oct 30 20:42:28.437483 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.437456    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-istio-envoy\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87\" (UID: \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"
Oct 30 20:42:28.437748 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.437724    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-istio-podinfo\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87\" (UID: \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"
Oct 30 20:42:28.495085 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.495011    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-istio-token\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87\" (UID: \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"
Oct 30 20:42:28.495085 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.495049    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdxfk\" (UniqueName: \"kubernetes.io/projected/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-kube-api-access-jdxfk\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87\" (UID: \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"
Oct 30 20:42:28.535725 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.535701    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"
Oct 30 20:42:28.618638 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.617934    5071 generic.go:356] "Generic (PLEG): container finished" podID="4036a76d-bdb3-4542-aae7-50d1bb377050" containerID="4c89174d9b13969cf4efe062cdb0446d2ce3bc1ba2e669ce557415b0e40ed794" exitCode=0
Oct 30 20:42:28.618638 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.618086    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1318e0ff-jwxgh" event={"ID":"4036a76d-bdb3-4542-aae7-50d1bb377050","Type":"ContainerDied","Data":"4c89174d9b13969cf4efe062cdb0446d2ce3bc1ba2e669ce557415b0e40ed794"}
Oct 30 20:42:28.622739 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.622708    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dd79b45-sshj7" event={"ID":"d7cd5a70-c322-4277-8a96-ebb7b539d088","Type":"ContainerStarted","Data":"d808682681f740430b61931999e763e8e3cd17f32a353aab2563b06172457771"}
Oct 30 20:42:28.622877 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.622765    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dd79b45-sshj7" event={"ID":"d7cd5a70-c322-4277-8a96-ebb7b539d088","Type":"ContainerStarted","Data":"1257aa73839f7a8135296afd404e4049087eee8677505987d8ca1fe8a64a946b"}
Oct 30 20:42:28.691401 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.691376    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"]
Oct 30 20:42:28.692907 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:42:28.692880    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05a72fc0_2554_4f73_935e_ade8fb962a7c.slice/crio-ada05eb7d0396ad1129ae1827bf3302b9163fc37ff9a7d791658cc4ed4eee994 WatchSource:0}: Error finding container ada05eb7d0396ad1129ae1827bf3302b9163fc37ff9a7d791658cc4ed4eee994: Status 404 returned error can't find the container with id ada05eb7d0396ad1129ae1827bf3302b9163fc37ff9a7d791658cc4ed4eee994
Oct 30 20:42:28.695353 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.695317    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:42:28.695441 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.695398    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:42:28.695441 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.695431    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:42:28.879395 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.879342    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dd79b45-sshj7" podStartSLOduration=2.879328022 podStartE2EDuration="2.879328022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:28.874523333 +0000 UTC m=+960.791544868" watchObservedRunningTime="2026-10-30 20:42:28.879328022 +0000 UTC m=+960.796349558"
Oct 30 20:42:28.929409 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:42:28.929379    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod723f1905_6d5a_45a7_a2bb_d5753a0fda9c.slice/crio-b320b360310298dbdfd63f7f280d9cd55bf22214be8a54bc3f411e759be94a60 WatchSource:0}: Error finding container b320b360310298dbdfd63f7f280d9cd55bf22214be8a54bc3f411e759be94a60: Status 404 returned error can't find the container with id b320b360310298dbdfd63f7f280d9cd55bf22214be8a54bc3f411e759be94a60
Oct 30 20:42:28.929527 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.929474    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"]
Oct 30 20:42:28.932291 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.932259    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:42:28.932381 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.932324    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:42:28.932381 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:28.932351    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:42:29.682047 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:29.682011    5071 generic.go:356] "Generic (PLEG): container finished" podID="d7cd5a70-c322-4277-8a96-ebb7b539d088" containerID="d808682681f740430b61931999e763e8e3cd17f32a353aab2563b06172457771" exitCode=0
Oct 30 20:42:29.682778 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:29.682748    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dd79b45-sshj7" event={"ID":"d7cd5a70-c322-4277-8a96-ebb7b539d088","Type":"ContainerDied","Data":"d808682681f740430b61931999e763e8e3cd17f32a353aab2563b06172457771"}
Oct 30 20:42:29.686844 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:29.686433    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87" event={"ID":"723f1905-6d5a-45a7-a2bb-d5753a0fda9c","Type":"ContainerStarted","Data":"b3689f02a0c436ed2add7966e749d0c6f7cc617b2bad25d8452f9e7bf29e240b"}
Oct 30 20:42:29.686844 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:29.686496    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87" event={"ID":"723f1905-6d5a-45a7-a2bb-d5753a0fda9c","Type":"ContainerStarted","Data":"b320b360310298dbdfd63f7f280d9cd55bf22214be8a54bc3f411e759be94a60"}
Oct 30 20:42:29.688198 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:29.688174    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g" event={"ID":"05a72fc0-2554-4f73-935e-ade8fb962a7c","Type":"ContainerStarted","Data":"0d30aebc1f53767b2a8c9a23e00404dd3b977c5126fc9e26b5bc757dd3361f91"}
Oct 30 20:42:29.688276 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:29.688222    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g" event={"ID":"05a72fc0-2554-4f73-935e-ade8fb962a7c","Type":"ContainerStarted","Data":"ada05eb7d0396ad1129ae1827bf3302b9163fc37ff9a7d791658cc4ed4eee994"}
Oct 30 20:42:30.101121 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:30.100107    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87" podStartSLOduration=2.100085081 podStartE2EDuration="2.100085081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:30.072673279 +0000 UTC m=+961.989694815" watchObservedRunningTime="2026-10-30 20:42:30.100085081 +0000 UTC m=+962.017106618"
Oct 30 20:42:30.106778 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:30.106750    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1318e0ff-jwxgh"
Oct 30 20:42:30.201340 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:30.201309    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4036a76d-bdb3-4542-aae7-50d1bb377050-kube-api-access-gh29f\" (UniqueName: \"kubernetes.io/projected/4036a76d-bdb3-4542-aae7-50d1bb377050-kube-api-access-gh29f\") pod \"4036a76d-bdb3-4542-aae7-50d1bb377050\" (UID: \"4036a76d-bdb3-4542-aae7-50d1bb377050\") "
Oct 30 20:42:30.201340 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:30.201345    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4036a76d-bdb3-4542-aae7-50d1bb377050-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4036a76d-bdb3-4542-aae7-50d1bb377050-maas-api-xks-ca\") pod \"4036a76d-bdb3-4542-aae7-50d1bb377050\" (UID: \"4036a76d-bdb3-4542-aae7-50d1bb377050\") "
Oct 30 20:42:30.201588 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:30.201411    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4036a76d-bdb3-4542-aae7-50d1bb377050-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4036a76d-bdb3-4542-aae7-50d1bb377050-maas-api-openshift-ca\") pod \"4036a76d-bdb3-4542-aae7-50d1bb377050\" (UID: \"4036a76d-bdb3-4542-aae7-50d1bb377050\") "
Oct 30 20:42:30.201884 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:30.201849    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4036a76d-bdb3-4542-aae7-50d1bb377050-maas-api-openshift-ca" pod "4036a76d-bdb3-4542-aae7-50d1bb377050" (UID: "4036a76d-bdb3-4542-aae7-50d1bb377050"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:42:30.203700 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:30.203675    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4036a76d-bdb3-4542-aae7-50d1bb377050-kube-api-access-gh29f" pod "4036a76d-bdb3-4542-aae7-50d1bb377050" (UID: "4036a76d-bdb3-4542-aae7-50d1bb377050"). InnerVolumeSpecName "kube-api-access-gh29f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:42:30.203970 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:30.203945    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4036a76d-bdb3-4542-aae7-50d1bb377050-maas-api-xks-ca" pod "4036a76d-bdb3-4542-aae7-50d1bb377050" (UID: "4036a76d-bdb3-4542-aae7-50d1bb377050"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:42:30.211202 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:30.211146    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g" podStartSLOduration=3.211128771 podStartE2EDuration="3.211128771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:30.209797809 +0000 UTC m=+962.126819347" watchObservedRunningTime="2026-10-30 20:42:30.211128771 +0000 UTC m=+962.128150308"
Oct 30 20:42:30.283905 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:30.283140    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"
Oct 30 20:42:30.283905 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:30.283366    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 20:42:30.283905 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:30.283438    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g" podUID="05a72fc0-2554-4f73-935e-ade8fb962a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 20:42:30.308845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:30.308209    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh29f\" (UniqueName: \"kubernetes.io/projected/4036a76d-bdb3-4542-aae7-50d1bb377050-kube-api-access-gh29f\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:30.308845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:30.308243    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4036a76d-bdb3-4542-aae7-50d1bb377050-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:30.308845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:30.308260    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4036a76d-bdb3-4542-aae7-50d1bb377050-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:30.542659 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:30.542614    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:30.555855 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:30.552174    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"
Oct 30 20:42:30.567960 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:30.567929    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 20:42:30.568204 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:30.568177    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87" podUID="723f1905-6d5a-45a7-a2bb-d5753a0fda9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 20:42:30.699467 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:30.699434    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1318e0ff-jwxgh" event={"ID":"4036a76d-bdb3-4542-aae7-50d1bb377050","Type":"ContainerDied","Data":"7c4d0e982428933396bce8e9573c1e312ce2d475af8bb23177bf7eafb2a1c9f2"}
Oct 30 20:42:30.700005 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:30.699480    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c4d0e982428933396bce8e9573c1e312ce2d475af8bb23177bf7eafb2a1c9f2"
Oct 30 20:42:30.700005 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:30.699919    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1318e0ff-jwxgh"
Oct 30 20:42:31.051849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:31.048880    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dd79b45-sshj7"
Oct 30 20:42:31.120893 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:31.119844    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7cd5a70-c322-4277-8a96-ebb7b539d088-kube-api-access-r84sc\" (UniqueName: \"kubernetes.io/projected/d7cd5a70-c322-4277-8a96-ebb7b539d088-kube-api-access-r84sc\") pod \"d7cd5a70-c322-4277-8a96-ebb7b539d088\" (UID: \"d7cd5a70-c322-4277-8a96-ebb7b539d088\") "
Oct 30 20:42:31.120893 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:31.120090    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7cd5a70-c322-4277-8a96-ebb7b539d088-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7cd5a70-c322-4277-8a96-ebb7b539d088-maas-api-openshift-ca\") pod \"d7cd5a70-c322-4277-8a96-ebb7b539d088\" (UID: \"d7cd5a70-c322-4277-8a96-ebb7b539d088\") "
Oct 30 20:42:31.120893 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:31.120189    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7cd5a70-c322-4277-8a96-ebb7b539d088-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7cd5a70-c322-4277-8a96-ebb7b539d088-maas-api-xks-ca\") pod \"d7cd5a70-c322-4277-8a96-ebb7b539d088\" (UID: \"d7cd5a70-c322-4277-8a96-ebb7b539d088\") "
Oct 30 20:42:31.121111 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:31.120901    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7cd5a70-c322-4277-8a96-ebb7b539d088-maas-api-openshift-ca" pod "d7cd5a70-c322-4277-8a96-ebb7b539d088" (UID: "d7cd5a70-c322-4277-8a96-ebb7b539d088"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:42:31.122801 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:31.122778    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7cd5a70-c322-4277-8a96-ebb7b539d088-kube-api-access-r84sc" pod "d7cd5a70-c322-4277-8a96-ebb7b539d088" (UID: "d7cd5a70-c322-4277-8a96-ebb7b539d088"). InnerVolumeSpecName "kube-api-access-r84sc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:42:31.125285 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:31.125266    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7cd5a70-c322-4277-8a96-ebb7b539d088-maas-api-xks-ca" pod "d7cd5a70-c322-4277-8a96-ebb7b539d088" (UID: "d7cd5a70-c322-4277-8a96-ebb7b539d088"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:42:31.221689 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:31.221647    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7cd5a70-c322-4277-8a96-ebb7b539d088-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:31.221689 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:31.221691    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r84sc\" (UniqueName: \"kubernetes.io/projected/d7cd5a70-c322-4277-8a96-ebb7b539d088-kube-api-access-r84sc\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:31.221918 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:31.221706    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7cd5a70-c322-4277-8a96-ebb7b539d088-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:31.282846 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:31.282223    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 20:42:31.282846 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:31.282291    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g" podUID="05a72fc0-2554-4f73-935e-ade8fb962a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 20:42:31.543849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:31.540198    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 20:42:31.543849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:31.540266    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87" podUID="723f1905-6d5a-45a7-a2bb-d5753a0fda9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 20:42:31.707536 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:31.707471    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dd79b45-sshj7" event={"ID":"d7cd5a70-c322-4277-8a96-ebb7b539d088","Type":"ContainerDied","Data":"1257aa73839f7a8135296afd404e4049087eee8677505987d8ca1fe8a64a946b"}
Oct 30 20:42:31.707536 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:31.707514    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dd79b45-sshj7"
Oct 30 20:42:31.708144 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:31.708076    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1257aa73839f7a8135296afd404e4049087eee8677505987d8ca1fe8a64a946b"
Oct 30 20:42:32.278847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:32.278224    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 20:42:32.278847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:32.278289    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g" podUID="05a72fc0-2554-4f73-935e-ade8fb962a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 20:42:32.548846 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:32.548079    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 20:42:32.548846 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:32.548139    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87" podUID="723f1905-6d5a-45a7-a2bb-d5753a0fda9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 20:42:33.278280 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:33.278239    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 20:42:33.279706 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:33.279670    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g" podUID="05a72fc0-2554-4f73-935e-ade8fb962a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 20:42:33.544913 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:33.536272    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 20:42:33.544913 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:33.536350    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87" podUID="723f1905-6d5a-45a7-a2bb-d5753a0fda9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 20:42:34.275300 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:34.275255    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 20:42:34.275613 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:34.275585    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g" podUID="05a72fc0-2554-4f73-935e-ade8fb962a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 20:42:34.540427 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:34.540393    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 20:42:34.541124 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:34.541073    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87" podUID="723f1905-6d5a-45a7-a2bb-d5753a0fda9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 20:42:35.275098 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:35.275054    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 20:42:35.275319 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:35.275127    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g" podUID="05a72fc0-2554-4f73-935e-ade8fb962a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 20:42:35.537234 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:35.537133    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 20:42:35.537234 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:35.537204    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87" podUID="723f1905-6d5a-45a7-a2bb-d5753a0fda9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 20:42:35.784933 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:35.784904    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:36.130638 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:36.130588    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:36.279335 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:36.279306    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"
Oct 30 20:42:36.279699 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:36.279676    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"
Oct 30 20:42:36.280423 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:36.280405    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"
Oct 30 20:42:36.540945 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:36.540916    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"
Oct 30 20:42:36.541226 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:36.541202    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"
Oct 30 20:42:36.541961 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:36.541941    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"
Oct 30 20:42:37.849844 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:37.849690    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d445009-xqn4p"]
Oct 30 20:42:37.853772 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:37.851294    5071 cpu_manager.go:414] "containerMap: removing container" podUID="4036a76d-bdb3-4542-aae7-50d1bb377050" containerName="revoke-keys"
Oct 30 20:42:37.853772 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:37.851324    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4036a76d-bdb3-4542-aae7-50d1bb377050" containerName="revoke-keys"
Oct 30 20:42:37.853772 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:37.851369    5071 cpu_manager.go:414] "containerMap: removing container" podUID="d7cd5a70-c322-4277-8a96-ebb7b539d088" containerName="revoke-keys"
Oct 30 20:42:37.853772 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:37.851376    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7cd5a70-c322-4277-8a96-ebb7b539d088" containerName="revoke-keys"
Oct 30 20:42:37.853772 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:37.851550    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="4036a76d-bdb3-4542-aae7-50d1bb377050" containerName="revoke-keys"
Oct 30 20:42:37.853772 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:37.851575    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7cd5a70-c322-4277-8a96-ebb7b539d088" containerName="revoke-keys"
Oct 30 20:42:37.862644 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:37.862603    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d445009-xqn4p"
Oct 30 20:42:37.922845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:37.917296    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa87e87b-e87a-4045-9266-7d4b720d14ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d445009-xqn4p\" (UID: \"aa87e87b-e87a-4045-9266-7d4b720d14ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d445009-xqn4p"
Oct 30 20:42:37.926894 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:37.924065    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa87e87b-e87a-4045-9266-7d4b720d14ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d445009-xqn4p\" (UID: \"aa87e87b-e87a-4045-9266-7d4b720d14ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d445009-xqn4p"
Oct 30 20:42:37.926894 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:37.924120    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vbll\" (UniqueName: \"kubernetes.io/projected/aa87e87b-e87a-4045-9266-7d4b720d14ea-kube-api-access-5vbll\") pod \"maas-api-revoke-sub-8d445009-xqn4p\" (UID: \"aa87e87b-e87a-4045-9266-7d4b720d14ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d445009-xqn4p"
Oct 30 20:42:37.926894 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:37.926707    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d445009-xqn4p"]
Oct 30 20:42:37.952893 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:37.952861    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:42:37.953294 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:37.953270    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 20:42:38.070167 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:38.070129    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa87e87b-e87a-4045-9266-7d4b720d14ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d445009-xqn4p\" (UID: \"aa87e87b-e87a-4045-9266-7d4b720d14ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d445009-xqn4p"
Oct 30 20:42:38.075932 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:38.075903    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa87e87b-e87a-4045-9266-7d4b720d14ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d445009-xqn4p\" (UID: \"aa87e87b-e87a-4045-9266-7d4b720d14ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d445009-xqn4p"
Oct 30 20:42:38.076544 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:38.076526    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vbll\" (UniqueName: \"kubernetes.io/projected/aa87e87b-e87a-4045-9266-7d4b720d14ea-kube-api-access-5vbll\") pod \"maas-api-revoke-sub-8d445009-xqn4p\" (UID: \"aa87e87b-e87a-4045-9266-7d4b720d14ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d445009-xqn4p"
Oct 30 20:42:38.077650 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:38.071628    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa87e87b-e87a-4045-9266-7d4b720d14ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d445009-xqn4p\" (UID: \"aa87e87b-e87a-4045-9266-7d4b720d14ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d445009-xqn4p"
Oct 30 20:42:38.119458 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:38.119380    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa87e87b-e87a-4045-9266-7d4b720d14ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d445009-xqn4p\" (UID: \"aa87e87b-e87a-4045-9266-7d4b720d14ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d445009-xqn4p"
Oct 30 20:42:38.139846 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:38.129710    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1976c9ab-phg6d"]
Oct 30 20:42:38.158587 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:38.158559    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1976c9ab-phg6d"
Oct 30 20:42:38.181578 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:38.180488    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vbll\" (UniqueName: \"kubernetes.io/projected/aa87e87b-e87a-4045-9266-7d4b720d14ea-kube-api-access-5vbll\") pod \"maas-api-revoke-sub-8d445009-xqn4p\" (UID: \"aa87e87b-e87a-4045-9266-7d4b720d14ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d445009-xqn4p"
Oct 30 20:42:38.181578 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:38.181236    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f47fddaf-9a6c-4d08-beb4-3bef431ee07b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1976c9ab-phg6d\" (UID: \"f47fddaf-9a6c-4d08-beb4-3bef431ee07b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1976c9ab-phg6d"
Oct 30 20:42:38.181578 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:38.181345    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcpd2\" (UniqueName: \"kubernetes.io/projected/f47fddaf-9a6c-4d08-beb4-3bef431ee07b-kube-api-access-dcpd2\") pod \"maas-api-revoke-sub-1976c9ab-phg6d\" (UID: \"f47fddaf-9a6c-4d08-beb4-3bef431ee07b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1976c9ab-phg6d"
Oct 30 20:42:38.181578 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:38.181439    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f47fddaf-9a6c-4d08-beb4-3bef431ee07b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1976c9ab-phg6d\" (UID: \"f47fddaf-9a6c-4d08-beb4-3bef431ee07b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1976c9ab-phg6d"
Oct 30 20:42:38.223673 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:38.223650    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d445009-xqn4p"
Oct 30 20:42:38.245852 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:38.242842    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1976c9ab-phg6d"]
Oct 30 20:42:38.320775 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:38.320744    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f47fddaf-9a6c-4d08-beb4-3bef431ee07b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1976c9ab-phg6d\" (UID: \"f47fddaf-9a6c-4d08-beb4-3bef431ee07b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1976c9ab-phg6d"
Oct 30 20:42:38.321095 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:38.321075    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f47fddaf-9a6c-4d08-beb4-3bef431ee07b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1976c9ab-phg6d\" (UID: \"f47fddaf-9a6c-4d08-beb4-3bef431ee07b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1976c9ab-phg6d"
Oct 30 20:42:38.321263 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:38.321248    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcpd2\" (UniqueName: \"kubernetes.io/projected/f47fddaf-9a6c-4d08-beb4-3bef431ee07b-kube-api-access-dcpd2\") pod \"maas-api-revoke-sub-1976c9ab-phg6d\" (UID: \"f47fddaf-9a6c-4d08-beb4-3bef431ee07b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1976c9ab-phg6d"
Oct 30 20:42:38.323016 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:38.322991    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f47fddaf-9a6c-4d08-beb4-3bef431ee07b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1976c9ab-phg6d\" (UID: \"f47fddaf-9a6c-4d08-beb4-3bef431ee07b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1976c9ab-phg6d"
Oct 30 20:42:38.379468 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:38.379307    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f47fddaf-9a6c-4d08-beb4-3bef431ee07b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1976c9ab-phg6d\" (UID: \"f47fddaf-9a6c-4d08-beb4-3bef431ee07b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1976c9ab-phg6d"
Oct 30 20:42:38.411369 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:38.411289    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcpd2\" (UniqueName: \"kubernetes.io/projected/f47fddaf-9a6c-4d08-beb4-3bef431ee07b-kube-api-access-dcpd2\") pod \"maas-api-revoke-sub-1976c9ab-phg6d\" (UID: \"f47fddaf-9a6c-4d08-beb4-3bef431ee07b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1976c9ab-phg6d"
Oct 30 20:42:38.666924 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:38.636556    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1976c9ab-phg6d"
Oct 30 20:42:38.680559 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:38.680474    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08207784-5kc2g"]
Oct 30 20:42:38.686676 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:38.686503    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08207784-5kc2g"
Oct 30 20:42:38.767093 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:38.762496    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08207784-5kc2g"]
Oct 30 20:42:38.767093 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:38.765296    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee132f80-47d9-49f2-ae94-b1d2b3c11c3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08207784-5kc2g\" (UID: \"ee132f80-47d9-49f2-ae94-b1d2b3c11c3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08207784-5kc2g"
Oct 30 20:42:38.767093 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:38.765364    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8db5\" (UniqueName: \"kubernetes.io/projected/ee132f80-47d9-49f2-ae94-b1d2b3c11c3a-kube-api-access-r8db5\") pod \"maas-api-revoke-sub-08207784-5kc2g\" (UID: \"ee132f80-47d9-49f2-ae94-b1d2b3c11c3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08207784-5kc2g"
Oct 30 20:42:38.767093 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:38.765445    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee132f80-47d9-49f2-ae94-b1d2b3c11c3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08207784-5kc2g\" (UID: \"ee132f80-47d9-49f2-ae94-b1d2b3c11c3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08207784-5kc2g"
Oct 30 20:42:38.863532 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:38.860048    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d445009-xqn4p"]
Oct 30 20:42:38.867040 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:38.866317    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee132f80-47d9-49f2-ae94-b1d2b3c11c3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08207784-5kc2g\" (UID: \"ee132f80-47d9-49f2-ae94-b1d2b3c11c3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08207784-5kc2g"
Oct 30 20:42:38.867040 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:38.866430    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee132f80-47d9-49f2-ae94-b1d2b3c11c3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08207784-5kc2g\" (UID: \"ee132f80-47d9-49f2-ae94-b1d2b3c11c3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08207784-5kc2g"
Oct 30 20:42:38.867040 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:38.866509    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8db5\" (UniqueName: \"kubernetes.io/projected/ee132f80-47d9-49f2-ae94-b1d2b3c11c3a-kube-api-access-r8db5\") pod \"maas-api-revoke-sub-08207784-5kc2g\" (UID: \"ee132f80-47d9-49f2-ae94-b1d2b3c11c3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08207784-5kc2g"
Oct 30 20:42:38.867442 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:38.867114    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee132f80-47d9-49f2-ae94-b1d2b3c11c3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08207784-5kc2g\" (UID: \"ee132f80-47d9-49f2-ae94-b1d2b3c11c3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08207784-5kc2g"
Oct 30 20:42:38.872753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:38.872211    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee132f80-47d9-49f2-ae94-b1d2b3c11c3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08207784-5kc2g\" (UID: \"ee132f80-47d9-49f2-ae94-b1d2b3c11c3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08207784-5kc2g"
Oct 30 20:42:38.925536 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:38.925510    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8db5\" (UniqueName: \"kubernetes.io/projected/ee132f80-47d9-49f2-ae94-b1d2b3c11c3a-kube-api-access-r8db5\") pod \"maas-api-revoke-sub-08207784-5kc2g\" (UID: \"ee132f80-47d9-49f2-ae94-b1d2b3c11c3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08207784-5kc2g"
Oct 30 20:42:39.006992 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:39.006966    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08207784-5kc2g"
Oct 30 20:42:39.114999 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:42:39.114967    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf47fddaf_9a6c_4d08_beb4_3bef431ee07b.slice/crio-8cc9df1b885a31f1f456b481ff2e59f50dbbcdd4aa225659228e2a60acc6849d WatchSource:0}: Error finding container 8cc9df1b885a31f1f456b481ff2e59f50dbbcdd4aa225659228e2a60acc6849d: Status 404 returned error can't find the container with id 8cc9df1b885a31f1f456b481ff2e59f50dbbcdd4aa225659228e2a60acc6849d
Oct 30 20:42:39.115413 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:39.115389    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1976c9ab-phg6d"]
Oct 30 20:42:39.396739 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:39.396715    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08207784-5kc2g"]
Oct 30 20:42:39.399392 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:42:39.399365    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee132f80_47d9_49f2_ae94_b1d2b3c11c3a.slice/crio-6542402c44ca078fb7edb639abe6e8347062c23cbc4148c32d9f8f0b89f73529 WatchSource:0}: Error finding container 6542402c44ca078fb7edb639abe6e8347062c23cbc4148c32d9f8f0b89f73529: Status 404 returned error can't find the container with id 6542402c44ca078fb7edb639abe6e8347062c23cbc4148c32d9f8f0b89f73529
Oct 30 20:42:39.810155 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:39.810122    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08207784-5kc2g" event={"ID":"ee132f80-47d9-49f2-ae94-b1d2b3c11c3a","Type":"ContainerStarted","Data":"6542402c44ca078fb7edb639abe6e8347062c23cbc4148c32d9f8f0b89f73529"}
Oct 30 20:42:39.811891 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:39.811858    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1976c9ab-phg6d" event={"ID":"f47fddaf-9a6c-4d08-beb4-3bef431ee07b","Type":"ContainerStarted","Data":"a27d74b5e77ecbf03cd816050d9ba51526ea55fdb55428abc2b8b4a4c86bc0b3"}
Oct 30 20:42:39.812042 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:39.811905    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1976c9ab-phg6d" event={"ID":"f47fddaf-9a6c-4d08-beb4-3bef431ee07b","Type":"ContainerStarted","Data":"8cc9df1b885a31f1f456b481ff2e59f50dbbcdd4aa225659228e2a60acc6849d"}
Oct 30 20:42:39.813376 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:39.813345    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d445009-xqn4p" event={"ID":"aa87e87b-e87a-4045-9266-7d4b720d14ea","Type":"ContainerStarted","Data":"1062ca2b6c343e9cc24e3c121c593c1d78ccbfe7fa3882af4157ba8a1bbd6797"}
Oct 30 20:42:39.813480 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:39.813383    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d445009-xqn4p" event={"ID":"aa87e87b-e87a-4045-9266-7d4b720d14ea","Type":"ContainerStarted","Data":"a0eea06a018643925ba9b395f6e2379ce90f788afd1c4b5012286d8876b2bc7c"}
Oct 30 20:42:39.943034 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:39.942942    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1976c9ab-phg6d" podStartSLOduration=1.9429269040000001 podStartE2EDuration="1.942926904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:39.942206481 +0000 UTC m=+971.859228017" watchObservedRunningTime="2026-10-30 20:42:39.942926904 +0000 UTC m=+971.859948477"
Oct 30 20:42:40.069082 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:40.069020    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d445009-xqn4p" podStartSLOduration=3.069002643 podStartE2EDuration="3.069002643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:40.067609959 +0000 UTC m=+971.984631495" watchObservedRunningTime="2026-10-30 20:42:40.069002643 +0000 UTC m=+971.986024182"
Oct 30 20:42:40.821621 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:40.821588    5071 generic.go:356] "Generic (PLEG): container finished" podID="f47fddaf-9a6c-4d08-beb4-3bef431ee07b" containerID="a27d74b5e77ecbf03cd816050d9ba51526ea55fdb55428abc2b8b4a4c86bc0b3" exitCode=0
Oct 30 20:42:40.821784 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:40.821678    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1976c9ab-phg6d" event={"ID":"f47fddaf-9a6c-4d08-beb4-3bef431ee07b","Type":"ContainerDied","Data":"a27d74b5e77ecbf03cd816050d9ba51526ea55fdb55428abc2b8b4a4c86bc0b3"}
Oct 30 20:42:40.823314 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:40.823283    5071 generic.go:356] "Generic (PLEG): container finished" podID="aa87e87b-e87a-4045-9266-7d4b720d14ea" containerID="1062ca2b6c343e9cc24e3c121c593c1d78ccbfe7fa3882af4157ba8a1bbd6797" exitCode=0
Oct 30 20:42:40.823448 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:40.823368    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d445009-xqn4p" event={"ID":"aa87e87b-e87a-4045-9266-7d4b720d14ea","Type":"ContainerDied","Data":"1062ca2b6c343e9cc24e3c121c593c1d78ccbfe7fa3882af4157ba8a1bbd6797"}
Oct 30 20:42:40.824995 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:40.824971    5071 generic.go:356] "Generic (PLEG): container finished" podID="ee132f80-47d9-49f2-ae94-b1d2b3c11c3a" containerID="ec4695415cdb07486a9b8b03e436548417b31ab4a9b6ee3cfc9f378d6a8e9f29" exitCode=0
Oct 30 20:42:40.825113 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:40.825044    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08207784-5kc2g" event={"ID":"ee132f80-47d9-49f2-ae94-b1d2b3c11c3a","Type":"ContainerDied","Data":"ec4695415cdb07486a9b8b03e436548417b31ab4a9b6ee3cfc9f378d6a8e9f29"}
Oct 30 20:42:41.008424 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:41.008390    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:42.051587 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.051557    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1976c9ab-phg6d"
Oct 30 20:42:42.102459 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.102435    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f47fddaf-9a6c-4d08-beb4-3bef431ee07b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f47fddaf-9a6c-4d08-beb4-3bef431ee07b-maas-api-openshift-ca\") pod \"f47fddaf-9a6c-4d08-beb4-3bef431ee07b\" (UID: \"f47fddaf-9a6c-4d08-beb4-3bef431ee07b\") "
Oct 30 20:42:42.102542 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.102527    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f47fddaf-9a6c-4d08-beb4-3bef431ee07b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f47fddaf-9a6c-4d08-beb4-3bef431ee07b-maas-api-xks-ca\") pod \"f47fddaf-9a6c-4d08-beb4-3bef431ee07b\" (UID: \"f47fddaf-9a6c-4d08-beb4-3bef431ee07b\") "
Oct 30 20:42:42.102618 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.102604    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f47fddaf-9a6c-4d08-beb4-3bef431ee07b-kube-api-access-dcpd2\" (UniqueName: \"kubernetes.io/projected/f47fddaf-9a6c-4d08-beb4-3bef431ee07b-kube-api-access-dcpd2\") pod \"f47fddaf-9a6c-4d08-beb4-3bef431ee07b\" (UID: \"f47fddaf-9a6c-4d08-beb4-3bef431ee07b\") "
Oct 30 20:42:42.102862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.102804    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f47fddaf-9a6c-4d08-beb4-3bef431ee07b-maas-api-openshift-ca" pod "f47fddaf-9a6c-4d08-beb4-3bef431ee07b" (UID: "f47fddaf-9a6c-4d08-beb4-3bef431ee07b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:42:42.103086 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.103060    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f47fddaf-9a6c-4d08-beb4-3bef431ee07b-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:42.105217 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.105179    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f47fddaf-9a6c-4d08-beb4-3bef431ee07b-maas-api-xks-ca" pod "f47fddaf-9a6c-4d08-beb4-3bef431ee07b" (UID: "f47fddaf-9a6c-4d08-beb4-3bef431ee07b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:42:42.105457 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.105430    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f47fddaf-9a6c-4d08-beb4-3bef431ee07b-kube-api-access-dcpd2" pod "f47fddaf-9a6c-4d08-beb4-3bef431ee07b" (UID: "f47fddaf-9a6c-4d08-beb4-3bef431ee07b"). InnerVolumeSpecName "kube-api-access-dcpd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:42:42.108129 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.108112    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08207784-5kc2g"
Oct 30 20:42:42.120316 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.120295    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d445009-xqn4p"
Oct 30 20:42:42.203870 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.203810    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa87e87b-e87a-4045-9266-7d4b720d14ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa87e87b-e87a-4045-9266-7d4b720d14ea-maas-api-xks-ca\") pod \"aa87e87b-e87a-4045-9266-7d4b720d14ea\" (UID: \"aa87e87b-e87a-4045-9266-7d4b720d14ea\") "
Oct 30 20:42:42.204031 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.203994    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee132f80-47d9-49f2-ae94-b1d2b3c11c3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee132f80-47d9-49f2-ae94-b1d2b3c11c3a-maas-api-openshift-ca\") pod \"ee132f80-47d9-49f2-ae94-b1d2b3c11c3a\" (UID: \"ee132f80-47d9-49f2-ae94-b1d2b3c11c3a\") "
Oct 30 20:42:42.204115 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.204097    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa87e87b-e87a-4045-9266-7d4b720d14ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa87e87b-e87a-4045-9266-7d4b720d14ea-maas-api-openshift-ca\") pod \"aa87e87b-e87a-4045-9266-7d4b720d14ea\" (UID: \"aa87e87b-e87a-4045-9266-7d4b720d14ea\") "
Oct 30 20:42:42.204178 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.204142    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee132f80-47d9-49f2-ae94-b1d2b3c11c3a-kube-api-access-r8db5\" (UniqueName: \"kubernetes.io/projected/ee132f80-47d9-49f2-ae94-b1d2b3c11c3a-kube-api-access-r8db5\") pod \"ee132f80-47d9-49f2-ae94-b1d2b3c11c3a\" (UID: \"ee132f80-47d9-49f2-ae94-b1d2b3c11c3a\") "
Oct 30 20:42:42.204178 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.204173    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee132f80-47d9-49f2-ae94-b1d2b3c11c3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee132f80-47d9-49f2-ae94-b1d2b3c11c3a-maas-api-xks-ca\") pod \"ee132f80-47d9-49f2-ae94-b1d2b3c11c3a\" (UID: \"ee132f80-47d9-49f2-ae94-b1d2b3c11c3a\") "
Oct 30 20:42:42.204290 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.204228    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa87e87b-e87a-4045-9266-7d4b720d14ea-kube-api-access-5vbll\" (UniqueName: \"kubernetes.io/projected/aa87e87b-e87a-4045-9266-7d4b720d14ea-kube-api-access-5vbll\") pod \"aa87e87b-e87a-4045-9266-7d4b720d14ea\" (UID: \"aa87e87b-e87a-4045-9266-7d4b720d14ea\") "
Oct 30 20:42:42.204472 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.204451    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee132f80-47d9-49f2-ae94-b1d2b3c11c3a-maas-api-openshift-ca" pod "ee132f80-47d9-49f2-ae94-b1d2b3c11c3a" (UID: "ee132f80-47d9-49f2-ae94-b1d2b3c11c3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:42:42.204586 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.204493    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa87e87b-e87a-4045-9266-7d4b720d14ea-maas-api-openshift-ca" pod "aa87e87b-e87a-4045-9266-7d4b720d14ea" (UID: "aa87e87b-e87a-4045-9266-7d4b720d14ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:42:42.204850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.204805    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f47fddaf-9a6c-4d08-beb4-3bef431ee07b-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:42.204954 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.204868    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee132f80-47d9-49f2-ae94-b1d2b3c11c3a-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:42.204954 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.204885    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcpd2\" (UniqueName: \"kubernetes.io/projected/f47fddaf-9a6c-4d08-beb4-3bef431ee07b-kube-api-access-dcpd2\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:42.204954 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.204901    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa87e87b-e87a-4045-9266-7d4b720d14ea-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:42.206345 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.206318    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa87e87b-e87a-4045-9266-7d4b720d14ea-maas-api-xks-ca" pod "aa87e87b-e87a-4045-9266-7d4b720d14ea" (UID: "aa87e87b-e87a-4045-9266-7d4b720d14ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:42:42.207307 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.207280    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee132f80-47d9-49f2-ae94-b1d2b3c11c3a-maas-api-xks-ca" pod "ee132f80-47d9-49f2-ae94-b1d2b3c11c3a" (UID: "ee132f80-47d9-49f2-ae94-b1d2b3c11c3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:42:42.208250 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.208227    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee132f80-47d9-49f2-ae94-b1d2b3c11c3a-kube-api-access-r8db5" pod "ee132f80-47d9-49f2-ae94-b1d2b3c11c3a" (UID: "ee132f80-47d9-49f2-ae94-b1d2b3c11c3a"). InnerVolumeSpecName "kube-api-access-r8db5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:42:42.208611 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.208585    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa87e87b-e87a-4045-9266-7d4b720d14ea-kube-api-access-5vbll" pod "aa87e87b-e87a-4045-9266-7d4b720d14ea" (UID: "aa87e87b-e87a-4045-9266-7d4b720d14ea"). InnerVolumeSpecName "kube-api-access-5vbll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:42:42.306529 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.306491    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee132f80-47d9-49f2-ae94-b1d2b3c11c3a-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:42.306529 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.306525    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vbll\" (UniqueName: \"kubernetes.io/projected/aa87e87b-e87a-4045-9266-7d4b720d14ea-kube-api-access-5vbll\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:42.306730 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.306540    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa87e87b-e87a-4045-9266-7d4b720d14ea-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:42.306730 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.306553    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8db5\" (UniqueName: \"kubernetes.io/projected/ee132f80-47d9-49f2-ae94-b1d2b3c11c3a-kube-api-access-r8db5\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:42.842304 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.842270    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08207784-5kc2g"
Oct 30 20:42:42.842304 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.842298    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08207784-5kc2g" event={"ID":"ee132f80-47d9-49f2-ae94-b1d2b3c11c3a","Type":"ContainerDied","Data":"6542402c44ca078fb7edb639abe6e8347062c23cbc4148c32d9f8f0b89f73529"}
Oct 30 20:42:42.842574 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.842377    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6542402c44ca078fb7edb639abe6e8347062c23cbc4148c32d9f8f0b89f73529"
Oct 30 20:42:42.844060 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.844035    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1976c9ab-phg6d"
Oct 30 20:42:42.844060 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.844059    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1976c9ab-phg6d" event={"ID":"f47fddaf-9a6c-4d08-beb4-3bef431ee07b","Type":"ContainerDied","Data":"8cc9df1b885a31f1f456b481ff2e59f50dbbcdd4aa225659228e2a60acc6849d"}
Oct 30 20:42:42.844270 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.844110    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cc9df1b885a31f1f456b481ff2e59f50dbbcdd4aa225659228e2a60acc6849d"
Oct 30 20:42:42.846176 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.846142    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d445009-xqn4p" event={"ID":"aa87e87b-e87a-4045-9266-7d4b720d14ea","Type":"ContainerDied","Data":"a0eea06a018643925ba9b395f6e2379ce90f788afd1c4b5012286d8876b2bc7c"}
Oct 30 20:42:42.846376 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.846200    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0eea06a018643925ba9b395f6e2379ce90f788afd1c4b5012286d8876b2bc7c"
Oct 30 20:42:42.846376 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:42.846161    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d445009-xqn4p"
Oct 30 20:42:46.121870 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:46.121783    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:46.727921 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:42:46.727887    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:42:51.224887 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:51.219923    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:42:55.706578 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:55.706543    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:42:56.318304 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:42:56.318258    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:01.288415 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:01.288381    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:04.204366 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:04.204323    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:04.379360 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:04.379331    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:06.285597 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:06.285568    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:11.289425 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:11.289382    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:16.217322 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:16.217290    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:16.729125 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:43:16.729086    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:43:21.291908 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:21.291865    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:26.007681 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:26.007644    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:26.299727 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:26.299690    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:26.671871 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:26.671758    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:28.078172 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:28.078137    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:31.272475 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:31.272439    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:32.205905 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:32.205875    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"]
Oct 30 20:43:32.206409 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:32.206397    5071 cpu_manager.go:414] "containerMap: removing container" podUID="aa87e87b-e87a-4045-9266-7d4b720d14ea" containerName="revoke-keys"
Oct 30 20:43:32.206452 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:32.206411    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa87e87b-e87a-4045-9266-7d4b720d14ea" containerName="revoke-keys"
Oct 30 20:43:32.206452 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:32.206422    5071 cpu_manager.go:414] "containerMap: removing container" podUID="f47fddaf-9a6c-4d08-beb4-3bef431ee07b" containerName="revoke-keys"
Oct 30 20:43:32.206452 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:32.206427    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f47fddaf-9a6c-4d08-beb4-3bef431ee07b" containerName="revoke-keys"
Oct 30 20:43:32.206452 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:32.206445    5071 cpu_manager.go:414] "containerMap: removing container" podUID="ee132f80-47d9-49f2-ae94-b1d2b3c11c3a" containerName="revoke-keys"
Oct 30 20:43:32.206452 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:32.206450    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee132f80-47d9-49f2-ae94-b1d2b3c11c3a" containerName="revoke-keys"
Oct 30 20:43:32.206606 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:32.206532    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa87e87b-e87a-4045-9266-7d4b720d14ea" containerName="revoke-keys"
Oct 30 20:43:32.206606 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:32.206540    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="f47fddaf-9a6c-4d08-beb4-3bef431ee07b" containerName="revoke-keys"
Oct 30 20:43:32.206606 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:32.206548    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee132f80-47d9-49f2-ae94-b1d2b3c11c3a" containerName="revoke-keys"
Oct 30 20:43:32.210070 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:32.210055    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:43:32.314303 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:32.314253    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b43c84\"/\"distinct-w2-b43c84-kserve-self-signed-certs\""
Oct 30 20:43:32.314849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:32.314378    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b43c84\"/\"openshift-service-ca.crt\""
Oct 30 20:43:32.314849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:32.314394    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b43c84\"/\"default-dockercfg-jgd8f\""
Oct 30 20:43:32.314849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:32.314452    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b43c84\"/\"kube-root-ca.crt\""
Oct 30 20:43:32.318265 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:32.318232    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"]
Oct 30 20:43:32.321108 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:32.321082    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-tmp-dir\") pod \"distinct-w2-b43c84-kserve-7765fd4d67-pcgfm\" (UID: \"5fbbb12c-4e14-45b1-a00e-84871cdcc5cc\") " pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:43:32.321245 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:32.321114    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-tls-certs\") pod \"distinct-w2-b43c84-kserve-7765fd4d67-pcgfm\" (UID: \"5fbbb12c-4e14-45b1-a00e-84871cdcc5cc\") " pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:43:32.321245 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:32.321139    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-home\") pod \"distinct-w2-b43c84-kserve-7765fd4d67-pcgfm\" (UID: \"5fbbb12c-4e14-45b1-a00e-84871cdcc5cc\") " pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:43:32.322695 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:32.322670    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-dshm\") pod \"distinct-w2-b43c84-kserve-7765fd4d67-pcgfm\" (UID: \"5fbbb12c-4e14-45b1-a00e-84871cdcc5cc\") " pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:43:32.322858 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:32.322782    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtxks\" (UniqueName: \"kubernetes.io/projected/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-kube-api-access-xtxks\") pod \"distinct-w2-b43c84-kserve-7765fd4d67-pcgfm\" (UID: \"5fbbb12c-4e14-45b1-a00e-84871cdcc5cc\") " pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:43:32.322858 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:32.322834    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-model-cache\") pod \"distinct-w2-b43c84-kserve-7765fd4d67-pcgfm\" (UID: \"5fbbb12c-4e14-45b1-a00e-84871cdcc5cc\") " pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:43:32.334535 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:32.334499    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:32.425744 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:32.425494    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-dshm\") pod \"distinct-w2-b43c84-kserve-7765fd4d67-pcgfm\" (UID: \"5fbbb12c-4e14-45b1-a00e-84871cdcc5cc\") " pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:43:32.425744 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:32.425607    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtxks\" (UniqueName: \"kubernetes.io/projected/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-kube-api-access-xtxks\") pod \"distinct-w2-b43c84-kserve-7765fd4d67-pcgfm\" (UID: \"5fbbb12c-4e14-45b1-a00e-84871cdcc5cc\") " pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:43:32.425744 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:32.425639    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-model-cache\") pod \"distinct-w2-b43c84-kserve-7765fd4d67-pcgfm\" (UID: \"5fbbb12c-4e14-45b1-a00e-84871cdcc5cc\") " pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:43:32.425744 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:32.425724    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-tmp-dir\") pod \"distinct-w2-b43c84-kserve-7765fd4d67-pcgfm\" (UID: \"5fbbb12c-4e14-45b1-a00e-84871cdcc5cc\") " pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:43:32.425744 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:32.425751    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-tls-certs\") pod \"distinct-w2-b43c84-kserve-7765fd4d67-pcgfm\" (UID: \"5fbbb12c-4e14-45b1-a00e-84871cdcc5cc\") " pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:43:32.426181 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:32.425779    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-home\") pod \"distinct-w2-b43c84-kserve-7765fd4d67-pcgfm\" (UID: \"5fbbb12c-4e14-45b1-a00e-84871cdcc5cc\") " pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:43:32.426899 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:32.426296    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-home\") pod \"distinct-w2-b43c84-kserve-7765fd4d67-pcgfm\" (UID: \"5fbbb12c-4e14-45b1-a00e-84871cdcc5cc\") " pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:43:32.450239 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:32.434703    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-dshm\") pod \"distinct-w2-b43c84-kserve-7765fd4d67-pcgfm\" (UID: \"5fbbb12c-4e14-45b1-a00e-84871cdcc5cc\") " pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:43:32.450239 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:32.435316    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-model-cache\") pod \"distinct-w2-b43c84-kserve-7765fd4d67-pcgfm\" (UID: \"5fbbb12c-4e14-45b1-a00e-84871cdcc5cc\") " pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:43:32.450239 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:32.435580    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-tmp-dir\") pod \"distinct-w2-b43c84-kserve-7765fd4d67-pcgfm\" (UID: \"5fbbb12c-4e14-45b1-a00e-84871cdcc5cc\") " pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:43:32.450239 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:32.441159    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-tls-certs\") pod \"distinct-w2-b43c84-kserve-7765fd4d67-pcgfm\" (UID: \"5fbbb12c-4e14-45b1-a00e-84871cdcc5cc\") " pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:43:32.494342 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:32.494272    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtxks\" (UniqueName: \"kubernetes.io/projected/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-kube-api-access-xtxks\") pod \"distinct-w2-b43c84-kserve-7765fd4d67-pcgfm\" (UID: \"5fbbb12c-4e14-45b1-a00e-84871cdcc5cc\") " pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:43:32.523301 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:32.523275    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:43:32.960624 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:43:32.960591    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fbbb12c_4e14_45b1_a00e_84871cdcc5cc.slice/crio-162a2c3347a858574d7ce38e7fd9936fb1a821b3a4c3326f8838a22fe208336c WatchSource:0}: Error finding container 162a2c3347a858574d7ce38e7fd9936fb1a821b3a4c3326f8838a22fe208336c: Status 404 returned error can't find the container with id 162a2c3347a858574d7ce38e7fd9936fb1a821b3a4c3326f8838a22fe208336c
Oct 30 20:43:32.960796 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:32.960772    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"]
Oct 30 20:43:33.226811 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:33.226716    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" event={"ID":"5fbbb12c-4e14-45b1-a00e-84871cdcc5cc","Type":"ContainerStarted","Data":"c5b2e09d90c044707594465e2eedcb3862b41c1fdd7797a39a4a70e9539f763b"}
Oct 30 20:43:33.226811 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:33.226760    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" event={"ID":"5fbbb12c-4e14-45b1-a00e-84871cdcc5cc","Type":"ContainerStarted","Data":"162a2c3347a858574d7ce38e7fd9936fb1a821b3a4c3326f8838a22fe208336c"}
Oct 30 20:43:33.344837 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:33.344773    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podStartSLOduration=1.3447546479999999 podStartE2EDuration="1.344754648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:43:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:43:33.342404388 +0000 UTC m=+1025.259425925" watchObservedRunningTime="2026-10-30 20:43:33.344754648 +0000 UTC m=+1025.261776188"
Oct 30 20:43:33.524352 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:33.524257    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:43:36.284949 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:36.284909    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:36.423609 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:36.423575    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:43:40.925918 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:40.925797    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:41.268155 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:41.268066    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:42.526559 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:42.526523    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:43:42.540499 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:42.540466    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:43:43.308649 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:43.308624    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:43:46.286058 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:46.286017    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:46.727481 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:43:46.727445    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:43:51.195536 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:51.195496    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:51.459152 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:51.459076    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:55.681680 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:55.681643    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"]
Oct 30 20:43:55.686010 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:55.685987    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:43:55.790112 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:55.790081    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-395022\"/\"distinct-w4-395022-kserve-self-signed-certs\""
Oct 30 20:43:55.793560 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:55.793540    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"]
Oct 30 20:43:55.797923 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:55.797883    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:55.815444 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:55.815401    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-model-cache\") pod \"distinct-w4-395022-kserve-77f46f9b8d-sn9tw\" (UID: \"1b9bf342-553e-4f8b-aa1c-71fd95b70b1f\") " pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:43:55.815751 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:55.815726    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp645\" (UniqueName: \"kubernetes.io/projected/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-kube-api-access-qp645\") pod \"distinct-w4-395022-kserve-77f46f9b8d-sn9tw\" (UID: \"1b9bf342-553e-4f8b-aa1c-71fd95b70b1f\") " pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:43:55.816047 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:55.816031    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-tmp-dir\") pod \"distinct-w4-395022-kserve-77f46f9b8d-sn9tw\" (UID: \"1b9bf342-553e-4f8b-aa1c-71fd95b70b1f\") " pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:43:55.816258 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:55.816243    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-tls-certs\") pod \"distinct-w4-395022-kserve-77f46f9b8d-sn9tw\" (UID: \"1b9bf342-553e-4f8b-aa1c-71fd95b70b1f\") " pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:43:55.816448 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:55.816430    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-dshm\") pod \"distinct-w4-395022-kserve-77f46f9b8d-sn9tw\" (UID: \"1b9bf342-553e-4f8b-aa1c-71fd95b70b1f\") " pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:43:55.816634 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:55.816619    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-home\") pod \"distinct-w4-395022-kserve-77f46f9b8d-sn9tw\" (UID: \"1b9bf342-553e-4f8b-aa1c-71fd95b70b1f\") " pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:43:55.917741 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:55.917709    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-tls-certs\") pod \"distinct-w4-395022-kserve-77f46f9b8d-sn9tw\" (UID: \"1b9bf342-553e-4f8b-aa1c-71fd95b70b1f\") " pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:43:55.917960 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:55.917744    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-dshm\") pod \"distinct-w4-395022-kserve-77f46f9b8d-sn9tw\" (UID: \"1b9bf342-553e-4f8b-aa1c-71fd95b70b1f\") " pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:43:55.917960 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:55.917786    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-home\") pod \"distinct-w4-395022-kserve-77f46f9b8d-sn9tw\" (UID: \"1b9bf342-553e-4f8b-aa1c-71fd95b70b1f\") " pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:43:55.917960 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:55.917868    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-model-cache\") pod \"distinct-w4-395022-kserve-77f46f9b8d-sn9tw\" (UID: \"1b9bf342-553e-4f8b-aa1c-71fd95b70b1f\") " pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:43:55.917960 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:55.917906    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp645\" (UniqueName: \"kubernetes.io/projected/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-kube-api-access-qp645\") pod \"distinct-w4-395022-kserve-77f46f9b8d-sn9tw\" (UID: \"1b9bf342-553e-4f8b-aa1c-71fd95b70b1f\") " pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:43:55.918201 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:55.918022    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-tmp-dir\") pod \"distinct-w4-395022-kserve-77f46f9b8d-sn9tw\" (UID: \"1b9bf342-553e-4f8b-aa1c-71fd95b70b1f\") " pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:43:55.918338 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:55.918318    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-home\") pod \"distinct-w4-395022-kserve-77f46f9b8d-sn9tw\" (UID: \"1b9bf342-553e-4f8b-aa1c-71fd95b70b1f\") " pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:43:55.918424 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:55.918403    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-model-cache\") pod \"distinct-w4-395022-kserve-77f46f9b8d-sn9tw\" (UID: \"1b9bf342-553e-4f8b-aa1c-71fd95b70b1f\") " pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:43:55.918500 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:55.918484    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-tmp-dir\") pod \"distinct-w4-395022-kserve-77f46f9b8d-sn9tw\" (UID: \"1b9bf342-553e-4f8b-aa1c-71fd95b70b1f\") " pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:43:55.920225 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:55.920203    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-dshm\") pod \"distinct-w4-395022-kserve-77f46f9b8d-sn9tw\" (UID: \"1b9bf342-553e-4f8b-aa1c-71fd95b70b1f\") " pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:43:55.920553 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:55.920533    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-tls-certs\") pod \"distinct-w4-395022-kserve-77f46f9b8d-sn9tw\" (UID: \"1b9bf342-553e-4f8b-aa1c-71fd95b70b1f\") " pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:43:55.976503 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:55.976434    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp645\" (UniqueName: \"kubernetes.io/projected/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-kube-api-access-qp645\") pod \"distinct-w4-395022-kserve-77f46f9b8d-sn9tw\" (UID: \"1b9bf342-553e-4f8b-aa1c-71fd95b70b1f\") " pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:43:55.999109 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:55.999083    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:43:56.112992 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:56.112944    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:43:56.378781 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:56.378753    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"]
Oct 30 20:43:56.380021 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:43:56.379997    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b9bf342_553e_4f8b_aa1c_71fd95b70b1f.slice/crio-b7984a9e21acbbc32eab71382f7bb3f0c154416471d9541766c34a250bb8aa8d WatchSource:0}: Error finding container b7984a9e21acbbc32eab71382f7bb3f0c154416471d9541766c34a250bb8aa8d: Status 404 returned error can't find the container with id b7984a9e21acbbc32eab71382f7bb3f0c154416471d9541766c34a250bb8aa8d
Oct 30 20:43:56.411679 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:56.411650    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" event={"ID":"1b9bf342-553e-4f8b-aa1c-71fd95b70b1f","Type":"ContainerStarted","Data":"b7984a9e21acbbc32eab71382f7bb3f0c154416471d9541766c34a250bb8aa8d"}
Oct 30 20:43:57.419954 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:57.419920    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" event={"ID":"1b9bf342-553e-4f8b-aa1c-71fd95b70b1f","Type":"ContainerStarted","Data":"b11501b47f868d4440aef56c6fa201025614da58454209e738244333f0bb8d42"}
Oct 30 20:43:57.543696 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:57.543636    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podStartSLOduration=2.5436208689999997 podStartE2EDuration="2.543620869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:43:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:43:57.539790574 +0000 UTC m=+1049.456812110" watchObservedRunningTime="2026-10-30 20:43:57.543620869 +0000 UTC m=+1049.460642415"
Oct 30 20:43:57.999646 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:43:57.999616    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:44:01.072667 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:01.072617    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:01.291377 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:01.291334    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:06.000087 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:06.000052    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:44:06.013592 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:06.013565    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:44:06.179241 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:06.179205    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:06.496472 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:06.496443    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:44:10.553944 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:10.553902    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:11.175113 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:11.175078    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:11.519363 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:11.519280    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:44:13.715048 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:13.715016    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-m6n4v-pdhcx"]
Oct 30 20:44:13.719046 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:13.719023    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-m6n4v-pdhcx"
Oct 30 20:44:13.777396 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:13.777360    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-m6n4v-pdhcx"]
Oct 30 20:44:13.825617 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:13.825591    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/95b944d4-560b-48b9-929e-df69e4d9a213-archives-path\") pod \"periodic-gathering-m6n4v-pdhcx\" (UID: \"95b944d4-560b-48b9-929e-df69e4d9a213\") " pod="openshift-insights/periodic-gathering-m6n4v-pdhcx"
Oct 30 20:44:13.825778 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:13.825631    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95b944d4-560b-48b9-929e-df69e4d9a213-trusted-ca-bundle\") pod \"periodic-gathering-m6n4v-pdhcx\" (UID: \"95b944d4-560b-48b9-929e-df69e4d9a213\") " pod="openshift-insights/periodic-gathering-m6n4v-pdhcx"
Oct 30 20:44:13.825778 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:13.825704    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cztlh\" (UniqueName: \"kubernetes.io/projected/95b944d4-560b-48b9-929e-df69e4d9a213-kube-api-access-cztlh\") pod \"periodic-gathering-m6n4v-pdhcx\" (UID: \"95b944d4-560b-48b9-929e-df69e4d9a213\") " pod="openshift-insights/periodic-gathering-m6n4v-pdhcx"
Oct 30 20:44:13.825778 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:13.825767    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95b944d4-560b-48b9-929e-df69e4d9a213-service-ca-bundle\") pod \"periodic-gathering-m6n4v-pdhcx\" (UID: \"95b944d4-560b-48b9-929e-df69e4d9a213\") " pod="openshift-insights/periodic-gathering-m6n4v-pdhcx"
Oct 30 20:44:13.927000 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:13.926970    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/95b944d4-560b-48b9-929e-df69e4d9a213-archives-path\") pod \"periodic-gathering-m6n4v-pdhcx\" (UID: \"95b944d4-560b-48b9-929e-df69e4d9a213\") " pod="openshift-insights/periodic-gathering-m6n4v-pdhcx"
Oct 30 20:44:13.927161 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:13.927012    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95b944d4-560b-48b9-929e-df69e4d9a213-trusted-ca-bundle\") pod \"periodic-gathering-m6n4v-pdhcx\" (UID: \"95b944d4-560b-48b9-929e-df69e4d9a213\") " pod="openshift-insights/periodic-gathering-m6n4v-pdhcx"
Oct 30 20:44:13.927161 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:13.927052    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cztlh\" (UniqueName: \"kubernetes.io/projected/95b944d4-560b-48b9-929e-df69e4d9a213-kube-api-access-cztlh\") pod \"periodic-gathering-m6n4v-pdhcx\" (UID: \"95b944d4-560b-48b9-929e-df69e4d9a213\") " pod="openshift-insights/periodic-gathering-m6n4v-pdhcx"
Oct 30 20:44:13.927161 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:13.927090    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95b944d4-560b-48b9-929e-df69e4d9a213-service-ca-bundle\") pod \"periodic-gathering-m6n4v-pdhcx\" (UID: \"95b944d4-560b-48b9-929e-df69e4d9a213\") " pod="openshift-insights/periodic-gathering-m6n4v-pdhcx"
Oct 30 20:44:13.927461 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:13.927433    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/95b944d4-560b-48b9-929e-df69e4d9a213-archives-path\") pod \"periodic-gathering-m6n4v-pdhcx\" (UID: \"95b944d4-560b-48b9-929e-df69e4d9a213\") " pod="openshift-insights/periodic-gathering-m6n4v-pdhcx"
Oct 30 20:44:13.927796 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:13.927773    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95b944d4-560b-48b9-929e-df69e4d9a213-trusted-ca-bundle\") pod \"periodic-gathering-m6n4v-pdhcx\" (UID: \"95b944d4-560b-48b9-929e-df69e4d9a213\") " pod="openshift-insights/periodic-gathering-m6n4v-pdhcx"
Oct 30 20:44:13.927914 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:13.927813    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95b944d4-560b-48b9-929e-df69e4d9a213-service-ca-bundle\") pod \"periodic-gathering-m6n4v-pdhcx\" (UID: \"95b944d4-560b-48b9-929e-df69e4d9a213\") " pod="openshift-insights/periodic-gathering-m6n4v-pdhcx"
Oct 30 20:44:13.985969 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:13.985901    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cztlh\" (UniqueName: \"kubernetes.io/projected/95b944d4-560b-48b9-929e-df69e4d9a213-kube-api-access-cztlh\") pod \"periodic-gathering-m6n4v-pdhcx\" (UID: \"95b944d4-560b-48b9-929e-df69e4d9a213\") " pod="openshift-insights/periodic-gathering-m6n4v-pdhcx"
Oct 30 20:44:14.033385 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:14.033355    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-m6n4v-pdhcx"
Oct 30 20:44:14.409982 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:14.409953    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-m6n4v-pdhcx"]
Oct 30 20:44:14.410116 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:44:14.410094    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b944d4_560b_48b9_929e_df69e4d9a213.slice/crio-f4d7fbed170f9485237f69dcf86297e9a94502e664ce198f508fcab7d61c455a WatchSource:0}: Error finding container f4d7fbed170f9485237f69dcf86297e9a94502e664ce198f508fcab7d61c455a: Status 404 returned error can't find the container with id f4d7fbed170f9485237f69dcf86297e9a94502e664ce198f508fcab7d61c455a
Oct 30 20:44:14.562874 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:14.562810    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-m6n4v-pdhcx" event={"ID":"95b944d4-560b-48b9-929e-df69e4d9a213","Type":"ContainerStarted","Data":"296bea8c06c7eb492b5310ef5b1b92190581241c499fcf7b5e8fcf0fb6051386"}
Oct 30 20:44:14.563058 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:14.562889    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-m6n4v-pdhcx" event={"ID":"95b944d4-560b-48b9-929e-df69e4d9a213","Type":"ContainerStarted","Data":"f4d7fbed170f9485237f69dcf86297e9a94502e664ce198f508fcab7d61c455a"}
Oct 30 20:44:14.679793 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:14.679699    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-m6n4v-pdhcx" podStartSLOduration=1.679681901 podStartE2EDuration="1.679681901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:14.677454406 +0000 UTC m=+1066.594475942" watchObservedRunningTime="2026-10-30 20:44:14.679681901 +0000 UTC m=+1066.596703439"
Oct 30 20:44:15.158456 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:15.158423    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"]
Oct 30 20:44:15.162766 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:15.162748    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"
Oct 30 20:44:15.243397 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:15.243364    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/191943aa-3324-4132-9edf-d2fbfc748b87-istio-data\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8\" (UID: \"191943aa-3324-4132-9edf-d2fbfc748b87\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"
Oct 30 20:44:15.243557 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:15.243420    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/191943aa-3324-4132-9edf-d2fbfc748b87-credential-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8\" (UID: \"191943aa-3324-4132-9edf-d2fbfc748b87\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"
Oct 30 20:44:15.243557 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:15.243512    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/191943aa-3324-4132-9edf-d2fbfc748b87-istio-podinfo\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8\" (UID: \"191943aa-3324-4132-9edf-d2fbfc748b87\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"
Oct 30 20:44:15.243631 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:15.243584    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/191943aa-3324-4132-9edf-d2fbfc748b87-istio-envoy\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8\" (UID: \"191943aa-3324-4132-9edf-d2fbfc748b87\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"
Oct 30 20:44:15.243671 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:15.243633    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/191943aa-3324-4132-9edf-d2fbfc748b87-workload-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8\" (UID: \"191943aa-3324-4132-9edf-d2fbfc748b87\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"
Oct 30 20:44:15.243735 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:15.243709    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/191943aa-3324-4132-9edf-d2fbfc748b87-workload-certs\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8\" (UID: \"191943aa-3324-4132-9edf-d2fbfc748b87\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"
Oct 30 20:44:15.243770 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:15.243755    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/191943aa-3324-4132-9edf-d2fbfc748b87-istio-token\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8\" (UID: \"191943aa-3324-4132-9edf-d2fbfc748b87\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"
Oct 30 20:44:15.243867 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:15.243844    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/191943aa-3324-4132-9edf-d2fbfc748b87-istiod-ca-cert\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8\" (UID: \"191943aa-3324-4132-9edf-d2fbfc748b87\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"
Oct 30 20:44:15.244006 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:15.243878    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gcvf\" (UniqueName: \"kubernetes.io/projected/191943aa-3324-4132-9edf-d2fbfc748b87-kube-api-access-7gcvf\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8\" (UID: \"191943aa-3324-4132-9edf-d2fbfc748b87\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"
Oct 30 20:44:15.272249 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:15.272221    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"]
Oct 30 20:44:15.345727 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:15.345693    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/191943aa-3324-4132-9edf-d2fbfc748b87-credential-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8\" (UID: \"191943aa-3324-4132-9edf-d2fbfc748b87\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"
Oct 30 20:44:15.345953 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:15.345745    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/191943aa-3324-4132-9edf-d2fbfc748b87-istio-podinfo\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8\" (UID: \"191943aa-3324-4132-9edf-d2fbfc748b87\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"
Oct 30 20:44:15.345953 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:15.345773    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/191943aa-3324-4132-9edf-d2fbfc748b87-istio-envoy\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8\" (UID: \"191943aa-3324-4132-9edf-d2fbfc748b87\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"
Oct 30 20:44:15.345953 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:15.345800    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/191943aa-3324-4132-9edf-d2fbfc748b87-workload-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8\" (UID: \"191943aa-3324-4132-9edf-d2fbfc748b87\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"
Oct 30 20:44:15.345953 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:15.345861    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/191943aa-3324-4132-9edf-d2fbfc748b87-workload-certs\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8\" (UID: \"191943aa-3324-4132-9edf-d2fbfc748b87\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"
Oct 30 20:44:15.345953 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:15.345888    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/191943aa-3324-4132-9edf-d2fbfc748b87-istio-token\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8\" (UID: \"191943aa-3324-4132-9edf-d2fbfc748b87\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"
Oct 30 20:44:15.346263 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:15.345958    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/191943aa-3324-4132-9edf-d2fbfc748b87-istiod-ca-cert\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8\" (UID: \"191943aa-3324-4132-9edf-d2fbfc748b87\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"
Oct 30 20:44:15.346263 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:15.345983    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gcvf\" (UniqueName: \"kubernetes.io/projected/191943aa-3324-4132-9edf-d2fbfc748b87-kube-api-access-7gcvf\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8\" (UID: \"191943aa-3324-4132-9edf-d2fbfc748b87\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"
Oct 30 20:44:15.346263 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:15.346188    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/191943aa-3324-4132-9edf-d2fbfc748b87-credential-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8\" (UID: \"191943aa-3324-4132-9edf-d2fbfc748b87\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"
Oct 30 20:44:15.346497 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:15.346472    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/191943aa-3324-4132-9edf-d2fbfc748b87-istio-data\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8\" (UID: \"191943aa-3324-4132-9edf-d2fbfc748b87\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"
Oct 30 20:44:15.346593 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:15.346556    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/191943aa-3324-4132-9edf-d2fbfc748b87-workload-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8\" (UID: \"191943aa-3324-4132-9edf-d2fbfc748b87\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"
Oct 30 20:44:15.346745 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:15.346715    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/191943aa-3324-4132-9edf-d2fbfc748b87-workload-certs\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8\" (UID: \"191943aa-3324-4132-9edf-d2fbfc748b87\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"
Oct 30 20:44:15.346857 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:15.346753    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/191943aa-3324-4132-9edf-d2fbfc748b87-istiod-ca-cert\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8\" (UID: \"191943aa-3324-4132-9edf-d2fbfc748b87\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"
Oct 30 20:44:15.347100 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:15.347078    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/191943aa-3324-4132-9edf-d2fbfc748b87-istio-data\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8\" (UID: \"191943aa-3324-4132-9edf-d2fbfc748b87\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"
Oct 30 20:44:15.348383 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:15.348364    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/191943aa-3324-4132-9edf-d2fbfc748b87-istio-envoy\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8\" (UID: \"191943aa-3324-4132-9edf-d2fbfc748b87\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"
Oct 30 20:44:15.348718 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:15.348698    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/191943aa-3324-4132-9edf-d2fbfc748b87-istio-podinfo\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8\" (UID: \"191943aa-3324-4132-9edf-d2fbfc748b87\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"
Oct 30 20:44:15.404642 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:15.404608    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/191943aa-3324-4132-9edf-d2fbfc748b87-istio-token\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8\" (UID: \"191943aa-3324-4132-9edf-d2fbfc748b87\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"
Oct 30 20:44:15.404781 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:15.404614    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gcvf\" (UniqueName: \"kubernetes.io/projected/191943aa-3324-4132-9edf-d2fbfc748b87-kube-api-access-7gcvf\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8\" (UID: \"191943aa-3324-4132-9edf-d2fbfc748b87\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"
Oct 30 20:44:15.476705 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:15.476616    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"
Oct 30 20:44:15.924545 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:15.924513    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"]
Oct 30 20:44:15.925064 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:44:15.925030    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod191943aa_3324_4132_9edf_d2fbfc748b87.slice/crio-11d149d931b39b83d6d7d44857eab0f9f05668ebb790fdf4279702dffba27698 WatchSource:0}: Error finding container 11d149d931b39b83d6d7d44857eab0f9f05668ebb790fdf4279702dffba27698: Status 404 returned error can't find the container with id 11d149d931b39b83d6d7d44857eab0f9f05668ebb790fdf4279702dffba27698
Oct 30 20:44:15.927571 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:15.927538    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:44:15.927626 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:15.927606    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:44:15.927664 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:15.927635    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:44:16.301715 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:16.301684    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:16.584360 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:16.584273    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8" event={"ID":"191943aa-3324-4132-9edf-d2fbfc748b87","Type":"ContainerStarted","Data":"98f0c3af6a8bcfce29ea2314e71d8449f351240bbbb1d90cad38aeec82338608"}
Oct 30 20:44:16.584360 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:16.584338    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8" event={"ID":"191943aa-3324-4132-9edf-d2fbfc748b87","Type":"ContainerStarted","Data":"11d149d931b39b83d6d7d44857eab0f9f05668ebb790fdf4279702dffba27698"}
Oct 30 20:44:16.727675 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:44:16.727644    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:44:16.755948 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:16.755895    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8" podStartSLOduration=1.755879864 podStartE2EDuration="1.755879864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:16.752222081 +0000 UTC m=+1068.669243621" watchObservedRunningTime="2026-10-30 20:44:16.755879864 +0000 UTC m=+1068.672901473"
Oct 30 20:44:17.477413 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:17.477377    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"
Oct 30 20:44:17.479420 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:17.479390    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 20:44:17.479537 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:17.479442    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8" podUID="191943aa-3324-4132-9edf-d2fbfc748b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 20:44:18.222126 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:18.222094    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:18.480897 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:18.479756    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 20:44:18.480897 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:18.480163    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8" podUID="191943aa-3324-4132-9edf-d2fbfc748b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 20:44:19.477047 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:19.477014    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 20:44:19.477213 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:19.477080    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8" podUID="191943aa-3324-4132-9edf-d2fbfc748b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 20:44:20.477224 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:20.477181    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 20:44:20.477663 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:20.477254    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8" podUID="191943aa-3324-4132-9edf-d2fbfc748b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 20:44:21.352848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:21.352592    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:21.477987 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:21.477951    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 20:44:21.478501 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:21.478004    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8" podUID="191943aa-3324-4132-9edf-d2fbfc748b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 20:44:22.477293 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:22.477262    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 20:44:22.477448 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:22.477325    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8" podUID="191943aa-3324-4132-9edf-d2fbfc748b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 20:44:23.481700 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:23.481672    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"
Oct 30 20:44:23.482122 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:23.482097    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"
Oct 30 20:44:23.482913 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:23.482895    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"
Oct 30 20:44:26.284419 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:26.284380    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:26.467720 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:26.467680    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:28.414472 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:28.414396    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:28.623858 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:28.623807    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"]
Oct 30 20:44:28.629226 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:28.629204    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"
Oct 30 20:44:28.652104 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:28.652075    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26c59b24-19e6-4911-a608-5cefe1ab8204-istio-podinfo\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk\" (UID: \"26c59b24-19e6-4911-a608-5cefe1ab8204\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"
Oct 30 20:44:28.652250 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:28.652125    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26c59b24-19e6-4911-a608-5cefe1ab8204-workload-certs\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk\" (UID: \"26c59b24-19e6-4911-a608-5cefe1ab8204\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"
Oct 30 20:44:28.652250 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:28.652234    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvlj4\" (UniqueName: \"kubernetes.io/projected/26c59b24-19e6-4911-a608-5cefe1ab8204-kube-api-access-mvlj4\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk\" (UID: \"26c59b24-19e6-4911-a608-5cefe1ab8204\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"
Oct 30 20:44:28.652363 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:28.652286    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26c59b24-19e6-4911-a608-5cefe1ab8204-istio-envoy\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk\" (UID: \"26c59b24-19e6-4911-a608-5cefe1ab8204\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"
Oct 30 20:44:28.652413 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:28.652352    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c59b24-19e6-4911-a608-5cefe1ab8204-workload-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk\" (UID: \"26c59b24-19e6-4911-a608-5cefe1ab8204\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"
Oct 30 20:44:28.652413 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:28.652391    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26c59b24-19e6-4911-a608-5cefe1ab8204-istio-token\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk\" (UID: \"26c59b24-19e6-4911-a608-5cefe1ab8204\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"
Oct 30 20:44:28.652510 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:28.652426    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c59b24-19e6-4911-a608-5cefe1ab8204-credential-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk\" (UID: \"26c59b24-19e6-4911-a608-5cefe1ab8204\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"
Oct 30 20:44:28.652510 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:28.652466    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26c59b24-19e6-4911-a608-5cefe1ab8204-istiod-ca-cert\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk\" (UID: \"26c59b24-19e6-4911-a608-5cefe1ab8204\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"
Oct 30 20:44:28.652510 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:28.652506    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26c59b24-19e6-4911-a608-5cefe1ab8204-istio-data\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk\" (UID: \"26c59b24-19e6-4911-a608-5cefe1ab8204\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"
Oct 30 20:44:28.686372 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:28.686284    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-86d620-openshift-default-dockercfg-b92ff\""
Oct 30 20:44:28.740604 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:28.740572    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"]
Oct 30 20:44:28.753956 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:28.753922    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26c59b24-19e6-4911-a608-5cefe1ab8204-istio-podinfo\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk\" (UID: \"26c59b24-19e6-4911-a608-5cefe1ab8204\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"
Oct 30 20:44:28.754104 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:28.753987    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26c59b24-19e6-4911-a608-5cefe1ab8204-workload-certs\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk\" (UID: \"26c59b24-19e6-4911-a608-5cefe1ab8204\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"
Oct 30 20:44:28.754104 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:28.754054    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvlj4\" (UniqueName: \"kubernetes.io/projected/26c59b24-19e6-4911-a608-5cefe1ab8204-kube-api-access-mvlj4\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk\" (UID: \"26c59b24-19e6-4911-a608-5cefe1ab8204\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"
Oct 30 20:44:28.754229 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:28.754106    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26c59b24-19e6-4911-a608-5cefe1ab8204-istio-envoy\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk\" (UID: \"26c59b24-19e6-4911-a608-5cefe1ab8204\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"
Oct 30 20:44:28.754229 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:28.754133    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c59b24-19e6-4911-a608-5cefe1ab8204-workload-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk\" (UID: \"26c59b24-19e6-4911-a608-5cefe1ab8204\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"
Oct 30 20:44:28.754229 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:28.754167    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26c59b24-19e6-4911-a608-5cefe1ab8204-istio-token\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk\" (UID: \"26c59b24-19e6-4911-a608-5cefe1ab8204\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"
Oct 30 20:44:28.754229 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:28.754215    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c59b24-19e6-4911-a608-5cefe1ab8204-credential-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk\" (UID: \"26c59b24-19e6-4911-a608-5cefe1ab8204\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"
Oct 30 20:44:28.754439 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:28.754261    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26c59b24-19e6-4911-a608-5cefe1ab8204-istiod-ca-cert\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk\" (UID: \"26c59b24-19e6-4911-a608-5cefe1ab8204\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"
Oct 30 20:44:28.754439 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:28.754307    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26c59b24-19e6-4911-a608-5cefe1ab8204-istio-data\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk\" (UID: \"26c59b24-19e6-4911-a608-5cefe1ab8204\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"
Oct 30 20:44:28.755122 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:28.755095    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c59b24-19e6-4911-a608-5cefe1ab8204-workload-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk\" (UID: \"26c59b24-19e6-4911-a608-5cefe1ab8204\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"
Oct 30 20:44:28.755282 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:28.755128    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26c59b24-19e6-4911-a608-5cefe1ab8204-istio-data\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk\" (UID: \"26c59b24-19e6-4911-a608-5cefe1ab8204\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"
Oct 30 20:44:28.755656 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:28.755635    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26c59b24-19e6-4911-a608-5cefe1ab8204-workload-certs\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk\" (UID: \"26c59b24-19e6-4911-a608-5cefe1ab8204\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"
Oct 30 20:44:28.755781 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:28.755663    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c59b24-19e6-4911-a608-5cefe1ab8204-credential-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk\" (UID: \"26c59b24-19e6-4911-a608-5cefe1ab8204\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"
Oct 30 20:44:28.756301 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:28.756278    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26c59b24-19e6-4911-a608-5cefe1ab8204-istiod-ca-cert\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk\" (UID: \"26c59b24-19e6-4911-a608-5cefe1ab8204\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"
Oct 30 20:44:28.758174 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:28.758149    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26c59b24-19e6-4911-a608-5cefe1ab8204-istio-envoy\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk\" (UID: \"26c59b24-19e6-4911-a608-5cefe1ab8204\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"
Oct 30 20:44:28.758971 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:28.758951    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26c59b24-19e6-4911-a608-5cefe1ab8204-istio-podinfo\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk\" (UID: \"26c59b24-19e6-4911-a608-5cefe1ab8204\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"
Oct 30 20:44:28.815145 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:28.815110    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvlj4\" (UniqueName: \"kubernetes.io/projected/26c59b24-19e6-4911-a608-5cefe1ab8204-kube-api-access-mvlj4\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk\" (UID: \"26c59b24-19e6-4911-a608-5cefe1ab8204\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"
Oct 30 20:44:28.815295 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:28.815193    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26c59b24-19e6-4911-a608-5cefe1ab8204-istio-token\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk\" (UID: \"26c59b24-19e6-4911-a608-5cefe1ab8204\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"
Oct 30 20:44:28.942878 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:28.942772    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"
Oct 30 20:44:29.376436 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:44:29.376403    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26c59b24_19e6_4911_a608_5cefe1ab8204.slice/crio-97866626ca1da893026e1346afda81b91867066f0ddf48cc2d3b8c5f8a837860 WatchSource:0}: Error finding container 97866626ca1da893026e1346afda81b91867066f0ddf48cc2d3b8c5f8a837860: Status 404 returned error can't find the container with id 97866626ca1da893026e1346afda81b91867066f0ddf48cc2d3b8c5f8a837860
Oct 30 20:44:29.376599 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.376443    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"]
Oct 30 20:44:29.379225 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.379192    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:44:29.379325 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.379258    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:44:29.379325 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.379287    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:44:29.587495 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.587460    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"]
Oct 30 20:44:29.591864 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.591840    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"
Oct 30 20:44:29.655237 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.655159    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"]
Oct 30 20:44:29.667185 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.667153    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-istio-envoy\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-v9744\" (UID: \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"
Oct 30 20:44:29.667351 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.667203    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-credential-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-v9744\" (UID: \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"
Oct 30 20:44:29.667351 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.667332    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-istio-data\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-v9744\" (UID: \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"
Oct 30 20:44:29.667465 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.667360    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-istiod-ca-cert\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-v9744\" (UID: \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"
Oct 30 20:44:29.667529 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.667475    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-workload-certs\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-v9744\" (UID: \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"
Oct 30 20:44:29.667591 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.667568    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-istio-token\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-v9744\" (UID: \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"
Oct 30 20:44:29.667645 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.667633    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-istio-podinfo\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-v9744\" (UID: \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"
Oct 30 20:44:29.667700 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.667663    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhq26\" (UniqueName: \"kubernetes.io/projected/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-kube-api-access-hhq26\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-v9744\" (UID: \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"
Oct 30 20:44:29.667752 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.667697    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-workload-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-v9744\" (UID: \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"
Oct 30 20:44:29.678202 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.678175    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk" event={"ID":"26c59b24-19e6-4911-a608-5cefe1ab8204","Type":"ContainerStarted","Data":"4a459da3a7bbe34a743f24736edce169c166367cd0f57ef85e1985652a3f14a9"}
Oct 30 20:44:29.678357 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.678224    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk" event={"ID":"26c59b24-19e6-4911-a608-5cefe1ab8204","Type":"ContainerStarted","Data":"97866626ca1da893026e1346afda81b91867066f0ddf48cc2d3b8c5f8a837860"}
Oct 30 20:44:29.769492 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.769461    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-istio-envoy\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-v9744\" (UID: \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"
Oct 30 20:44:29.769492 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.769497    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-credential-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-v9744\" (UID: \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"
Oct 30 20:44:29.769720 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.769538    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-istio-data\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-v9744\" (UID: \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"
Oct 30 20:44:29.769720 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.769555    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-istiod-ca-cert\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-v9744\" (UID: \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"
Oct 30 20:44:29.769994 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.769964    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-credential-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-v9744\" (UID: \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"
Oct 30 20:44:29.770091 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.770019    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-istio-data\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-v9744\" (UID: \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"
Oct 30 20:44:29.770091 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.770044    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-workload-certs\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-v9744\" (UID: \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"
Oct 30 20:44:29.770370 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.770238    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-istiod-ca-cert\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-v9744\" (UID: \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"
Oct 30 20:44:29.770370 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.769634    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-workload-certs\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-v9744\" (UID: \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"
Oct 30 20:44:29.770370 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.770323    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-istio-token\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-v9744\" (UID: \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"
Oct 30 20:44:29.770370 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.770362    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-istio-podinfo\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-v9744\" (UID: \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"
Oct 30 20:44:29.770583 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.770380    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhq26\" (UniqueName: \"kubernetes.io/projected/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-kube-api-access-hhq26\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-v9744\" (UID: \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"
Oct 30 20:44:29.770583 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.770407    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-workload-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-v9744\" (UID: \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"
Oct 30 20:44:29.770932 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.770905    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-workload-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-v9744\" (UID: \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"
Oct 30 20:44:29.772127 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.772102    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-istio-envoy\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-v9744\" (UID: \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"
Oct 30 20:44:29.772914 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.772895    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-istio-podinfo\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-v9744\" (UID: \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"
Oct 30 20:44:29.830066 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.829672    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhq26\" (UniqueName: \"kubernetes.io/projected/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-kube-api-access-hhq26\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-v9744\" (UID: \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"
Oct 30 20:44:29.830345 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.830296    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-istio-token\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-v9744\" (UID: \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"
Oct 30 20:44:29.835864 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.835798    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk" podStartSLOduration=1.8357869629999999 podStartE2EDuration="1.835786963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:29.831290524 +0000 UTC m=+1081.748312059" watchObservedRunningTime="2026-10-30 20:44:29.835786963 +0000 UTC m=+1081.752808498"
Oct 30 20:44:29.907686 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.907599    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"]
Oct 30 20:44:29.909050 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.909024    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"
Oct 30 20:44:29.913219 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.913203    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"
Oct 30 20:44:29.956002 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.955946    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"
Oct 30 20:44:29.974816 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.972665    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98829221-3804-4f1f-9abe-24101fb2a2a0-credential-socket\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874\" (UID: \"98829221-3804-4f1f-9abe-24101fb2a2a0\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"
Oct 30 20:44:29.974816 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.972720    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98829221-3804-4f1f-9abe-24101fb2a2a0-workload-socket\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874\" (UID: \"98829221-3804-4f1f-9abe-24101fb2a2a0\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"
Oct 30 20:44:29.974816 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.972842    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98829221-3804-4f1f-9abe-24101fb2a2a0-istio-data\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874\" (UID: \"98829221-3804-4f1f-9abe-24101fb2a2a0\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"
Oct 30 20:44:29.974816 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.972877    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98829221-3804-4f1f-9abe-24101fb2a2a0-istio-token\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874\" (UID: \"98829221-3804-4f1f-9abe-24101fb2a2a0\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"
Oct 30 20:44:29.974816 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.972903    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98829221-3804-4f1f-9abe-24101fb2a2a0-workload-certs\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874\" (UID: \"98829221-3804-4f1f-9abe-24101fb2a2a0\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"
Oct 30 20:44:29.974816 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.972920    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns95h\" (UniqueName: \"kubernetes.io/projected/98829221-3804-4f1f-9abe-24101fb2a2a0-kube-api-access-ns95h\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874\" (UID: \"98829221-3804-4f1f-9abe-24101fb2a2a0\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"
Oct 30 20:44:29.974816 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.972955    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98829221-3804-4f1f-9abe-24101fb2a2a0-istio-envoy\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874\" (UID: \"98829221-3804-4f1f-9abe-24101fb2a2a0\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"
Oct 30 20:44:29.974816 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.972971    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98829221-3804-4f1f-9abe-24101fb2a2a0-istiod-ca-cert\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874\" (UID: \"98829221-3804-4f1f-9abe-24101fb2a2a0\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"
Oct 30 20:44:29.974816 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.973036    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98829221-3804-4f1f-9abe-24101fb2a2a0-istio-podinfo\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874\" (UID: \"98829221-3804-4f1f-9abe-24101fb2a2a0\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"
Oct 30 20:44:29.974816 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:29.974679    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"]
Oct 30 20:44:30.074809 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:30.074773    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98829221-3804-4f1f-9abe-24101fb2a2a0-istio-data\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874\" (UID: \"98829221-3804-4f1f-9abe-24101fb2a2a0\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"
Oct 30 20:44:30.075000 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:30.074857    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98829221-3804-4f1f-9abe-24101fb2a2a0-istio-token\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874\" (UID: \"98829221-3804-4f1f-9abe-24101fb2a2a0\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"
Oct 30 20:44:30.075000 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:30.074891    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98829221-3804-4f1f-9abe-24101fb2a2a0-workload-certs\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874\" (UID: \"98829221-3804-4f1f-9abe-24101fb2a2a0\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"
Oct 30 20:44:30.075000 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:30.074914    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns95h\" (UniqueName: \"kubernetes.io/projected/98829221-3804-4f1f-9abe-24101fb2a2a0-kube-api-access-ns95h\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874\" (UID: \"98829221-3804-4f1f-9abe-24101fb2a2a0\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"
Oct 30 20:44:30.075000 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:30.074968    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98829221-3804-4f1f-9abe-24101fb2a2a0-istio-envoy\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874\" (UID: \"98829221-3804-4f1f-9abe-24101fb2a2a0\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"
Oct 30 20:44:30.075000 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:30.074990    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98829221-3804-4f1f-9abe-24101fb2a2a0-istiod-ca-cert\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874\" (UID: \"98829221-3804-4f1f-9abe-24101fb2a2a0\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"
Oct 30 20:44:30.075161 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:30.075093    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98829221-3804-4f1f-9abe-24101fb2a2a0-istio-podinfo\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874\" (UID: \"98829221-3804-4f1f-9abe-24101fb2a2a0\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"
Oct 30 20:44:30.075215 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:30.075185    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98829221-3804-4f1f-9abe-24101fb2a2a0-credential-socket\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874\" (UID: \"98829221-3804-4f1f-9abe-24101fb2a2a0\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"
Oct 30 20:44:30.075978 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:30.075946    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98829221-3804-4f1f-9abe-24101fb2a2a0-workload-socket\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874\" (UID: \"98829221-3804-4f1f-9abe-24101fb2a2a0\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"
Oct 30 20:44:30.076123 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:30.075953    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98829221-3804-4f1f-9abe-24101fb2a2a0-istio-data\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874\" (UID: \"98829221-3804-4f1f-9abe-24101fb2a2a0\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"
Oct 30 20:44:30.076545 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:30.076244    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98829221-3804-4f1f-9abe-24101fb2a2a0-workload-certs\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874\" (UID: \"98829221-3804-4f1f-9abe-24101fb2a2a0\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"
Oct 30 20:44:30.076545 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:30.076478    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98829221-3804-4f1f-9abe-24101fb2a2a0-workload-socket\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874\" (UID: \"98829221-3804-4f1f-9abe-24101fb2a2a0\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"
Oct 30 20:44:30.076545 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:30.076513    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98829221-3804-4f1f-9abe-24101fb2a2a0-credential-socket\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874\" (UID: \"98829221-3804-4f1f-9abe-24101fb2a2a0\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"
Oct 30 20:44:30.076747 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:30.076695    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98829221-3804-4f1f-9abe-24101fb2a2a0-istiod-ca-cert\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874\" (UID: \"98829221-3804-4f1f-9abe-24101fb2a2a0\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"
Oct 30 20:44:30.079115 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:30.078566    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98829221-3804-4f1f-9abe-24101fb2a2a0-istio-envoy\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874\" (UID: \"98829221-3804-4f1f-9abe-24101fb2a2a0\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"
Oct 30 20:44:30.079115 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:30.078706    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98829221-3804-4f1f-9abe-24101fb2a2a0-istio-podinfo\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874\" (UID: \"98829221-3804-4f1f-9abe-24101fb2a2a0\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"
Oct 30 20:44:30.156590 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:30.156556    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98829221-3804-4f1f-9abe-24101fb2a2a0-istio-token\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874\" (UID: \"98829221-3804-4f1f-9abe-24101fb2a2a0\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"
Oct 30 20:44:30.156768 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:30.156748    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns95h\" (UniqueName: \"kubernetes.io/projected/98829221-3804-4f1f-9abe-24101fb2a2a0-kube-api-access-ns95h\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874\" (UID: \"98829221-3804-4f1f-9abe-24101fb2a2a0\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"
Oct 30 20:44:30.242173 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:30.242072    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"
Oct 30 20:44:30.360387 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:30.360356    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"]
Oct 30 20:44:30.364768 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:44:30.364743    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84cb8a9e_b4d1_4c1b_9338_2ea77abec886.slice/crio-4f9b850766e21965d77f426829fd998a8475f48968ced41a429163d9af36ec58 WatchSource:0}: Error finding container 4f9b850766e21965d77f426829fd998a8475f48968ced41a429163d9af36ec58: Status 404 returned error can't find the container with id 4f9b850766e21965d77f426829fd998a8475f48968ced41a429163d9af36ec58
Oct 30 20:44:30.368251 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:30.367857    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:44:30.368251 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:30.367948    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:44:30.368251 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:30.367993    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:44:30.645007 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:30.643900    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"]
Oct 30 20:44:30.666857 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:30.664467    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:44:30.666857 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:30.664593    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:44:30.666857 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:30.664636    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:44:30.721688 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:30.721657    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744" event={"ID":"84cb8a9e-b4d1-4c1b-9338-2ea77abec886","Type":"ContainerStarted","Data":"01f8afa45021133acda78d054117a722b14b334a8ed629371c812a4748bf5a3d"}
Oct 30 20:44:30.721876 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:30.721714    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744" event={"ID":"84cb8a9e-b4d1-4c1b-9338-2ea77abec886","Type":"ContainerStarted","Data":"4f9b850766e21965d77f426829fd998a8475f48968ced41a429163d9af36ec58"}
Oct 30 20:44:30.723681 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:30.723650    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874" event={"ID":"98829221-3804-4f1f-9abe-24101fb2a2a0","Type":"ContainerStarted","Data":"82f2b353c2604d370254d7d634a76ef1dcd527bcf26e80b1d6fcde2cc8c6ac0f"}
Oct 30 20:44:30.873849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:30.872083    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744" podStartSLOduration=1.872044678 podStartE2EDuration="1.872044678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:30.869257907 +0000 UTC m=+1082.786279450" watchObservedRunningTime="2026-10-30 20:44:30.872044678 +0000 UTC m=+1082.789066220"
Oct 30 20:44:30.910928 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:30.909607    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"
Oct 30 20:44:30.966946 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:30.955941    5071 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 20:44:30.966946 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:30.956003    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk" podUID="26c59b24-19e6-4911-a608-5cefe1ab8204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 20:44:31.481851 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:31.480814    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:31.757905 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:31.753884    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874" event={"ID":"98829221-3804-4f1f-9abe-24101fb2a2a0","Type":"ContainerStarted","Data":"06eac27309063b9e1c9d873460dd1f9edd4732e62a78372e90a7b19c945cff74"}
Oct 30 20:44:31.900852 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:31.900473    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874" podStartSLOduration=2.900449718 podStartE2EDuration="2.900449718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:31.881323418 +0000 UTC m=+1083.798344954" watchObservedRunningTime="2026-10-30 20:44:31.900449718 +0000 UTC m=+1083.817471255"
Oct 30 20:44:31.921856 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:31.920955    5071 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-v9744 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 20:44:31.921856 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:31.921024    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744" podUID="84cb8a9e-b4d1-4c1b-9338-2ea77abec886" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 20:44:31.944380 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:31.944255    5071 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 20:44:31.944380 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:31.944319    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk" podUID="26c59b24-19e6-4911-a608-5cefe1ab8204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 20:44:32.258863 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:32.258036    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"
Oct 30 20:44:32.272749 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:32.269977    5071 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 20:44:32.272749 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:32.270076    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874" podUID="98829221-3804-4f1f-9abe-24101fb2a2a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 20:44:32.910843 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:32.909862    5071 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-v9744 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 20:44:32.910843 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:32.909911    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744" podUID="84cb8a9e-b4d1-4c1b-9338-2ea77abec886" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 20:44:32.947849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:32.945173    5071 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 20:44:32.947849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:32.945233    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk" podUID="26c59b24-19e6-4911-a608-5cefe1ab8204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 20:44:33.253907 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:33.251190    5071 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 20:44:33.253907 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:33.251249    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874" podUID="98829221-3804-4f1f-9abe-24101fb2a2a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 20:44:33.910040 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:33.910008    5071 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-v9744 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 20:44:33.910276 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:33.910248    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744" podUID="84cb8a9e-b4d1-4c1b-9338-2ea77abec886" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 20:44:33.947861 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:33.945203    5071 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 20:44:33.947861 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:33.945267    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk" podUID="26c59b24-19e6-4911-a608-5cefe1ab8204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 20:44:34.265905 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:34.257219    5071 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 20:44:34.265905 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:34.257281    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874" podUID="98829221-3804-4f1f-9abe-24101fb2a2a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 20:44:34.913176 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:34.913139    5071 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-v9744 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 20:44:34.914132 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:34.914098    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744" podUID="84cb8a9e-b4d1-4c1b-9338-2ea77abec886" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 20:44:34.948261 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:34.948227    5071 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 20:44:34.948872 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:34.948811    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk" podUID="26c59b24-19e6-4911-a608-5cefe1ab8204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 20:44:35.258538 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:35.258434    5071 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 20:44:35.258847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:35.258786    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874" podUID="98829221-3804-4f1f-9abe-24101fb2a2a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 20:44:35.909749 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:35.909442    5071 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-v9744 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 20:44:35.909749 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:35.909513    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744" podUID="84cb8a9e-b4d1-4c1b-9338-2ea77abec886" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 20:44:35.947846 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:35.947197    5071 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 20:44:35.947846 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:35.947256    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk" podUID="26c59b24-19e6-4911-a608-5cefe1ab8204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 20:44:36.254901 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:36.254225    5071 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 20:44:36.254901 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:36.254288    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874" podUID="98829221-3804-4f1f-9abe-24101fb2a2a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 20:44:36.909575 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:36.909541    5071 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-v9744 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 20:44:36.909891 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:36.909595    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744" podUID="84cb8a9e-b4d1-4c1b-9338-2ea77abec886" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 20:44:37.004246 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:37.004216    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"
Oct 30 20:44:37.004912 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:37.004891    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"
Oct 30 20:44:37.006762 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:37.006745    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"
Oct 30 20:44:37.128981 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:37.127929    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:37.250601 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:37.250513    5071 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 20:44:37.250601 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:37.250570    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874" podUID="98829221-3804-4f1f-9abe-24101fb2a2a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 20:44:37.963848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:37.962839    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"
Oct 30 20:44:37.964303 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:37.963903    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"
Oct 30 20:44:37.964915 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:37.964897    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"
Oct 30 20:44:38.267193 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:38.267112    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"
Oct 30 20:44:38.269072 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:38.269046    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"
Oct 30 20:44:38.270023 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:38.270001    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"
Oct 30 20:44:42.226860 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:42.226792    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:46.728871 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:44:46.728803    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:44:47.101746 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:47.101710    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:52.014739 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:52.014698    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:44:55.618539 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:55.618503    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:44:56.908863 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:44:56.908798    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:00.185690 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.185657    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889885-5lxxx"]
Oct 30 20:45:00.191535 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.191511    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889885-cvfkd"]
Oct 30 20:45:00.191700 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.191637    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889885-5lxxx"
Oct 30 20:45:00.198976 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.198950    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889885-6nmnq"]
Oct 30 20:45:00.199101 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.199054    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889885-cvfkd"
Oct 30 20:45:00.202813 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.202794    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-6nmnq"
Oct 30 20:45:00.234745 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.234713    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f6ebf5-29889885-k6svg"]
Oct 30 20:45:00.238834 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.238798    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f6ebf5-29889885-k6svg"
Oct 30 20:45:00.245481 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.245352    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 20:45:00.292939 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.292899    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889885-cvfkd"]
Oct 30 20:45:00.296924 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.296894    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889885-5lxxx"]
Oct 30 20:45:00.345256 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.345226    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f6ebf5-29889885-k6svg"]
Oct 30 20:45:00.365133 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.365104    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26471d92-815c-471c-b8be-52f4fe79dbc8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-386743-29889885-5lxxx\" (UID: \"26471d92-815c-471c-b8be-52f4fe79dbc8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889885-5lxxx"
Oct 30 20:45:00.365375 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.365181    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ee26fb5-c5d7-4b70-a994-67b1c487e5da-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889885-6nmnq\" (UID: \"4ee26fb5-c5d7-4b70-a994-67b1c487e5da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-6nmnq"
Oct 30 20:45:00.365375 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.365255    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7g59\" (UniqueName: \"kubernetes.io/projected/26471d92-815c-471c-b8be-52f4fe79dbc8-kube-api-access-q7g59\") pod \"maas-api-key-cleanup-e2e-worker-w5-386743-29889885-5lxxx\" (UID: \"26471d92-815c-471c-b8be-52f4fe79dbc8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889885-5lxxx"
Oct 30 20:45:00.365375 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.365275    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2dv9\" (UniqueName: \"kubernetes.io/projected/4ee26fb5-c5d7-4b70-a994-67b1c487e5da-kube-api-access-t2dv9\") pod \"maas-api-key-cleanup-29889885-6nmnq\" (UID: \"4ee26fb5-c5d7-4b70-a994-67b1c487e5da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-6nmnq"
Oct 30 20:45:00.365375 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.365295    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3892545f-5bd2-490b-bd69-b1e8434da80c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f6ebf5-29889885-k6svg\" (UID: \"3892545f-5bd2-490b-bd69-b1e8434da80c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f6ebf5-29889885-k6svg"
Oct 30 20:45:00.365375 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.365356    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxx4b\" (UniqueName: \"kubernetes.io/projected/3892545f-5bd2-490b-bd69-b1e8434da80c-kube-api-access-rxx4b\") pod \"maas-api-key-cleanup-e2e-worker-w3-f6ebf5-29889885-k6svg\" (UID: \"3892545f-5bd2-490b-bd69-b1e8434da80c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f6ebf5-29889885-k6svg"
Oct 30 20:45:00.365375 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.365373    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spgn2\" (UniqueName: \"kubernetes.io/projected/0729ed1c-bbfd-48a8-97aa-f2fe75735a9e-kube-api-access-spgn2\") pod \"maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889885-cvfkd\" (UID: \"0729ed1c-bbfd-48a8-97aa-f2fe75735a9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889885-cvfkd"
Oct 30 20:45:00.365720 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.365453    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0729ed1c-bbfd-48a8-97aa-f2fe75735a9e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889885-cvfkd\" (UID: \"0729ed1c-bbfd-48a8-97aa-f2fe75735a9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889885-cvfkd"
Oct 30 20:45:00.394322 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.394291    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889885-6nmnq"]
Oct 30 20:45:00.466605 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.466514    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26471d92-815c-471c-b8be-52f4fe79dbc8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-386743-29889885-5lxxx\" (UID: \"26471d92-815c-471c-b8be-52f4fe79dbc8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889885-5lxxx"
Oct 30 20:45:00.466605 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.466596    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ee26fb5-c5d7-4b70-a994-67b1c487e5da-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889885-6nmnq\" (UID: \"4ee26fb5-c5d7-4b70-a994-67b1c487e5da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-6nmnq"
Oct 30 20:45:00.466869 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.466648    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7g59\" (UniqueName: \"kubernetes.io/projected/26471d92-815c-471c-b8be-52f4fe79dbc8-kube-api-access-q7g59\") pod \"maas-api-key-cleanup-e2e-worker-w5-386743-29889885-5lxxx\" (UID: \"26471d92-815c-471c-b8be-52f4fe79dbc8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889885-5lxxx"
Oct 30 20:45:00.466869 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.466681    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2dv9\" (UniqueName: \"kubernetes.io/projected/4ee26fb5-c5d7-4b70-a994-67b1c487e5da-kube-api-access-t2dv9\") pod \"maas-api-key-cleanup-29889885-6nmnq\" (UID: \"4ee26fb5-c5d7-4b70-a994-67b1c487e5da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-6nmnq"
Oct 30 20:45:00.466869 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.466711    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3892545f-5bd2-490b-bd69-b1e8434da80c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f6ebf5-29889885-k6svg\" (UID: \"3892545f-5bd2-490b-bd69-b1e8434da80c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f6ebf5-29889885-k6svg"
Oct 30 20:45:00.466869 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.466801    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxx4b\" (UniqueName: \"kubernetes.io/projected/3892545f-5bd2-490b-bd69-b1e8434da80c-kube-api-access-rxx4b\") pod \"maas-api-key-cleanup-e2e-worker-w3-f6ebf5-29889885-k6svg\" (UID: \"3892545f-5bd2-490b-bd69-b1e8434da80c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f6ebf5-29889885-k6svg"
Oct 30 20:45:00.467097 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.466868    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spgn2\" (UniqueName: \"kubernetes.io/projected/0729ed1c-bbfd-48a8-97aa-f2fe75735a9e-kube-api-access-spgn2\") pod \"maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889885-cvfkd\" (UID: \"0729ed1c-bbfd-48a8-97aa-f2fe75735a9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889885-cvfkd"
Oct 30 20:45:00.467097 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.466996    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0729ed1c-bbfd-48a8-97aa-f2fe75735a9e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889885-cvfkd\" (UID: \"0729ed1c-bbfd-48a8-97aa-f2fe75735a9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889885-cvfkd"
Oct 30 20:45:00.467392 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.467367    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26471d92-815c-471c-b8be-52f4fe79dbc8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-386743-29889885-5lxxx\" (UID: \"26471d92-815c-471c-b8be-52f4fe79dbc8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889885-5lxxx"
Oct 30 20:45:00.467456 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.467412    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ee26fb5-c5d7-4b70-a994-67b1c487e5da-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889885-6nmnq\" (UID: \"4ee26fb5-c5d7-4b70-a994-67b1c487e5da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-6nmnq"
Oct 30 20:45:00.467632 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.467613    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0729ed1c-bbfd-48a8-97aa-f2fe75735a9e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889885-cvfkd\" (UID: \"0729ed1c-bbfd-48a8-97aa-f2fe75735a9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889885-cvfkd"
Oct 30 20:45:00.467748 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.467728    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3892545f-5bd2-490b-bd69-b1e8434da80c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f6ebf5-29889885-k6svg\" (UID: \"3892545f-5bd2-490b-bd69-b1e8434da80c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f6ebf5-29889885-k6svg"
Oct 30 20:45:00.525948 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.525915    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxx4b\" (UniqueName: \"kubernetes.io/projected/3892545f-5bd2-490b-bd69-b1e8434da80c-kube-api-access-rxx4b\") pod \"maas-api-key-cleanup-e2e-worker-w3-f6ebf5-29889885-k6svg\" (UID: \"3892545f-5bd2-490b-bd69-b1e8434da80c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f6ebf5-29889885-k6svg"
Oct 30 20:45:00.526326 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.526300    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spgn2\" (UniqueName: \"kubernetes.io/projected/0729ed1c-bbfd-48a8-97aa-f2fe75735a9e-kube-api-access-spgn2\") pod \"maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889885-cvfkd\" (UID: \"0729ed1c-bbfd-48a8-97aa-f2fe75735a9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889885-cvfkd"
Oct 30 20:45:00.526377 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.526315    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7g59\" (UniqueName: \"kubernetes.io/projected/26471d92-815c-471c-b8be-52f4fe79dbc8-kube-api-access-q7g59\") pod \"maas-api-key-cleanup-e2e-worker-w5-386743-29889885-5lxxx\" (UID: \"26471d92-815c-471c-b8be-52f4fe79dbc8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889885-5lxxx"
Oct 30 20:45:00.526529 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.526508    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2dv9\" (UniqueName: \"kubernetes.io/projected/4ee26fb5-c5d7-4b70-a994-67b1c487e5da-kube-api-access-t2dv9\") pod \"maas-api-key-cleanup-29889885-6nmnq\" (UID: \"4ee26fb5-c5d7-4b70-a994-67b1c487e5da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-6nmnq"
Oct 30 20:45:00.552362 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.552331    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f6ebf5-29889885-k6svg"
Oct 30 20:45:00.809712 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.809684    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889885-5lxxx"
Oct 30 20:45:00.818014 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.817987    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889885-cvfkd"
Oct 30 20:45:00.823500 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.823479    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-6nmnq"
Oct 30 20:45:00.939925 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:45:00.939898    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3892545f_5bd2_490b_bd69_b1e8434da80c.slice/crio-a31246f96ccaa9aab31c4098fb04f8a6a52159162927d41a6f2ffd8979ccfdd0 WatchSource:0}: Error finding container a31246f96ccaa9aab31c4098fb04f8a6a52159162927d41a6f2ffd8979ccfdd0: Status 404 returned error can't find the container with id a31246f96ccaa9aab31c4098fb04f8a6a52159162927d41a6f2ffd8979ccfdd0
Oct 30 20:45:00.940421 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.940398    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f6ebf5-29889885-k6svg"]
Oct 30 20:45:00.981932 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.981871    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"]
Oct 30 20:45:00.989637 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:00.989613    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"
Oct 30 20:45:01.050426 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.050397    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"]
Oct 30 20:45:01.058151 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.058115    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f6ebf5-29889885-k6svg" event={"ID":"3892545f-5bd2-490b-bd69-b1e8434da80c","Type":"ContainerStarted","Data":"a31246f96ccaa9aab31c4098fb04f8a6a52159162927d41a6f2ffd8979ccfdd0"}
Oct 30 20:45:01.075313 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.075192    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/203f757a-7115-466c-8a13-14fb326599ca-istio-data\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz\" (UID: \"203f757a-7115-466c-8a13-14fb326599ca\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"
Oct 30 20:45:01.075313 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.075261    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/203f757a-7115-466c-8a13-14fb326599ca-workload-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz\" (UID: \"203f757a-7115-466c-8a13-14fb326599ca\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"
Oct 30 20:45:01.075583 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.075326    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/203f757a-7115-466c-8a13-14fb326599ca-istio-envoy\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz\" (UID: \"203f757a-7115-466c-8a13-14fb326599ca\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"
Oct 30 20:45:01.075583 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.075410    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/203f757a-7115-466c-8a13-14fb326599ca-istiod-ca-cert\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz\" (UID: \"203f757a-7115-466c-8a13-14fb326599ca\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"
Oct 30 20:45:01.075583 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.075454    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/203f757a-7115-466c-8a13-14fb326599ca-workload-certs\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz\" (UID: \"203f757a-7115-466c-8a13-14fb326599ca\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"
Oct 30 20:45:01.075583 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.075484    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/203f757a-7115-466c-8a13-14fb326599ca-credential-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz\" (UID: \"203f757a-7115-466c-8a13-14fb326599ca\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"
Oct 30 20:45:01.075583 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.075515    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/203f757a-7115-466c-8a13-14fb326599ca-istio-podinfo\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz\" (UID: \"203f757a-7115-466c-8a13-14fb326599ca\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"
Oct 30 20:45:01.075583 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.075575    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/203f757a-7115-466c-8a13-14fb326599ca-istio-token\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz\" (UID: \"203f757a-7115-466c-8a13-14fb326599ca\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"
Oct 30 20:45:01.075972 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.075662    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgl6v\" (UniqueName: \"kubernetes.io/projected/203f757a-7115-466c-8a13-14fb326599ca-kube-api-access-lgl6v\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz\" (UID: \"203f757a-7115-466c-8a13-14fb326599ca\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"
Oct 30 20:45:01.177383 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.177353    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/203f757a-7115-466c-8a13-14fb326599ca-istio-envoy\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz\" (UID: \"203f757a-7115-466c-8a13-14fb326599ca\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"
Oct 30 20:45:01.177565 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.177417    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/203f757a-7115-466c-8a13-14fb326599ca-istiod-ca-cert\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz\" (UID: \"203f757a-7115-466c-8a13-14fb326599ca\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"
Oct 30 20:45:01.177565 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.177461    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/203f757a-7115-466c-8a13-14fb326599ca-workload-certs\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz\" (UID: \"203f757a-7115-466c-8a13-14fb326599ca\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"
Oct 30 20:45:01.177565 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.177483    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/203f757a-7115-466c-8a13-14fb326599ca-credential-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz\" (UID: \"203f757a-7115-466c-8a13-14fb326599ca\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"
Oct 30 20:45:01.177565 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.177503    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/203f757a-7115-466c-8a13-14fb326599ca-istio-podinfo\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz\" (UID: \"203f757a-7115-466c-8a13-14fb326599ca\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"
Oct 30 20:45:01.177565 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.177556    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/203f757a-7115-466c-8a13-14fb326599ca-istio-token\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz\" (UID: \"203f757a-7115-466c-8a13-14fb326599ca\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"
Oct 30 20:45:01.178298 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.177638    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgl6v\" (UniqueName: \"kubernetes.io/projected/203f757a-7115-466c-8a13-14fb326599ca-kube-api-access-lgl6v\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz\" (UID: \"203f757a-7115-466c-8a13-14fb326599ca\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"
Oct 30 20:45:01.178298 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.177748    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/203f757a-7115-466c-8a13-14fb326599ca-istio-data\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz\" (UID: \"203f757a-7115-466c-8a13-14fb326599ca\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"
Oct 30 20:45:01.178298 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.177795    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/203f757a-7115-466c-8a13-14fb326599ca-workload-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz\" (UID: \"203f757a-7115-466c-8a13-14fb326599ca\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"
Oct 30 20:45:01.178298 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.178133    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/203f757a-7115-466c-8a13-14fb326599ca-credential-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz\" (UID: \"203f757a-7115-466c-8a13-14fb326599ca\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"
Oct 30 20:45:01.178549 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.178467    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/203f757a-7115-466c-8a13-14fb326599ca-istiod-ca-cert\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz\" (UID: \"203f757a-7115-466c-8a13-14fb326599ca\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"
Oct 30 20:45:01.178697 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.178662    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/203f757a-7115-466c-8a13-14fb326599ca-workload-certs\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz\" (UID: \"203f757a-7115-466c-8a13-14fb326599ca\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"
Oct 30 20:45:01.178883 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.178858    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/203f757a-7115-466c-8a13-14fb326599ca-workload-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz\" (UID: \"203f757a-7115-466c-8a13-14fb326599ca\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"
Oct 30 20:45:01.179129 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.178989    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/203f757a-7115-466c-8a13-14fb326599ca-istio-data\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz\" (UID: \"203f757a-7115-466c-8a13-14fb326599ca\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"
Oct 30 20:45:01.180692 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.180672    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/203f757a-7115-466c-8a13-14fb326599ca-istio-envoy\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz\" (UID: \"203f757a-7115-466c-8a13-14fb326599ca\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"
Oct 30 20:45:01.182164 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.182132    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/203f757a-7115-466c-8a13-14fb326599ca-istio-podinfo\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz\" (UID: \"203f757a-7115-466c-8a13-14fb326599ca\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"
Oct 30 20:45:01.201160 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.201133    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889885-5lxxx"]
Oct 30 20:45:01.201529 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:45:01.201404    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26471d92_815c_471c_b8be_52f4fe79dbc8.slice/crio-d9a8ba11734ea15547b260006522457deeb0562926b41e6f7b05629fae097fe6 WatchSource:0}: Error finding container d9a8ba11734ea15547b260006522457deeb0562926b41e6f7b05629fae097fe6: Status 404 returned error can't find the container with id d9a8ba11734ea15547b260006522457deeb0562926b41e6f7b05629fae097fe6
Oct 30 20:45:01.224938 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:45:01.224903    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0729ed1c_bbfd_48a8_97aa_f2fe75735a9e.slice/crio-d0066ce8ecb0301daa1a8f6433756f6280f53383c10bf84dd793886b09b4ca88 WatchSource:0}: Error finding container d0066ce8ecb0301daa1a8f6433756f6280f53383c10bf84dd793886b09b4ca88: Status 404 returned error can't find the container with id d0066ce8ecb0301daa1a8f6433756f6280f53383c10bf84dd793886b09b4ca88
Oct 30 20:45:01.225084 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.225060    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889885-cvfkd"]
Oct 30 20:45:01.236847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.236807    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgl6v\" (UniqueName: \"kubernetes.io/projected/203f757a-7115-466c-8a13-14fb326599ca-kube-api-access-lgl6v\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz\" (UID: \"203f757a-7115-466c-8a13-14fb326599ca\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"
Oct 30 20:45:01.237103 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.237083    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/203f757a-7115-466c-8a13-14fb326599ca-istio-token\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz\" (UID: \"203f757a-7115-466c-8a13-14fb326599ca\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"
Oct 30 20:45:01.249106 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:45:01.249074    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ee26fb5_c5d7_4b70_a994_67b1c487e5da.slice/crio-ff6f093390b39bf4112806264a259dd288d528aee79a51732d80b0cbe400a46c WatchSource:0}: Error finding container ff6f093390b39bf4112806264a259dd288d528aee79a51732d80b0cbe400a46c: Status 404 returned error can't find the container with id ff6f093390b39bf4112806264a259dd288d528aee79a51732d80b0cbe400a46c
Oct 30 20:45:01.250614 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.250587    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889885-6nmnq"]
Oct 30 20:45:01.305835 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.305785    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"
Oct 30 20:45:01.700335 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.700301    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"]
Oct 30 20:45:01.700471 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:45:01.700299    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod203f757a_7115_466c_8a13_14fb326599ca.slice/crio-75324956feb95a2ef538374c564fe4ec246f9452d263e35b0e1b3c4ea6f6ae5a WatchSource:0}: Error finding container 75324956feb95a2ef538374c564fe4ec246f9452d263e35b0e1b3c4ea6f6ae5a: Status 404 returned error can't find the container with id 75324956feb95a2ef538374c564fe4ec246f9452d263e35b0e1b3c4ea6f6ae5a
Oct 30 20:45:01.703146 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.703108    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:45:01.703265 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.703187    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:45:01.703265 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.703232    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:45:01.808794 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:01.808753    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:02.069251 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:02.069209    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-6nmnq" event={"ID":"4ee26fb5-c5d7-4b70-a994-67b1c487e5da","Type":"ContainerStarted","Data":"c178d91d805867b1f5206cb8611a197aa4e43bfa5b655eca41df61d48834dfea"}
Oct 30 20:45:02.069453 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:02.069283    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-6nmnq" event={"ID":"4ee26fb5-c5d7-4b70-a994-67b1c487e5da","Type":"ContainerStarted","Data":"ff6f093390b39bf4112806264a259dd288d528aee79a51732d80b0cbe400a46c"}
Oct 30 20:45:02.071449 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:02.071415    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz" event={"ID":"203f757a-7115-466c-8a13-14fb326599ca","Type":"ContainerStarted","Data":"bdbe5346a4464e79bf2af7ce1592376856c73ec10df12e85f0b66ddb403ab5cc"}
Oct 30 20:45:02.071567 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:02.071471    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz" event={"ID":"203f757a-7115-466c-8a13-14fb326599ca","Type":"ContainerStarted","Data":"75324956feb95a2ef538374c564fe4ec246f9452d263e35b0e1b3c4ea6f6ae5a"}
Oct 30 20:45:02.073730 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:02.073700    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889885-cvfkd" event={"ID":"0729ed1c-bbfd-48a8-97aa-f2fe75735a9e","Type":"ContainerStarted","Data":"cc7470c9993b97ebd555c5a326f03cf4dea437c25c095144dc4856e6e575ab34"}
Oct 30 20:45:02.073898 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:02.073747    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889885-cvfkd" event={"ID":"0729ed1c-bbfd-48a8-97aa-f2fe75735a9e","Type":"ContainerStarted","Data":"d0066ce8ecb0301daa1a8f6433756f6280f53383c10bf84dd793886b09b4ca88"}
Oct 30 20:45:02.075497 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:02.075473    5071 generic.go:356] "Generic (PLEG): container finished" podID="3892545f-5bd2-490b-bd69-b1e8434da80c" containerID="527eccfd8d6d9fb9fec90883c366b626a83290a3514604a7b9d3487d14ccd86d" exitCode=60
Oct 30 20:45:02.075621 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:02.075532    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f6ebf5-29889885-k6svg" event={"ID":"3892545f-5bd2-490b-bd69-b1e8434da80c","Type":"ContainerDied","Data":"527eccfd8d6d9fb9fec90883c366b626a83290a3514604a7b9d3487d14ccd86d"}
Oct 30 20:45:02.075810 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:02.075793    5071 scope.go:122] "RemoveContainer" containerID="527eccfd8d6d9fb9fec90883c366b626a83290a3514604a7b9d3487d14ccd86d"
Oct 30 20:45:02.077274 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:02.077248    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889885-5lxxx" event={"ID":"26471d92-815c-471c-b8be-52f4fe79dbc8","Type":"ContainerStarted","Data":"6a975508fae807349c96461b66baf3b33ee8c7bf0a67cf843d88f79451a7e05d"}
Oct 30 20:45:02.077368 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:02.077292    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889885-5lxxx" event={"ID":"26471d92-815c-471c-b8be-52f4fe79dbc8","Type":"ContainerStarted","Data":"d9a8ba11734ea15547b260006522457deeb0562926b41e6f7b05629fae097fe6"}
Oct 30 20:45:02.187114 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:02.187055    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-6nmnq" podStartSLOduration=2.187040522 podStartE2EDuration="2.187040522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:02.185626546 +0000 UTC m=+1114.102648092" watchObservedRunningTime="2026-10-30 20:45:02.187040522 +0000 UTC m=+1114.104062098"
Oct 30 20:45:02.301513 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:02.301454    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889885-cvfkd" podStartSLOduration=2.30143934 podStartE2EDuration="2.30143934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:02.300237057 +0000 UTC m=+1114.217258594" watchObservedRunningTime="2026-10-30 20:45:02.30143934 +0000 UTC m=+1114.218460906"
Oct 30 20:45:02.306274 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:02.306245    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"
Oct 30 20:45:02.308245 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:02.308215    5071 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 20:45:02.308380 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:02.308281    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz" podUID="203f757a-7115-466c-8a13-14fb326599ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 20:45:02.564656 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:02.563513    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz" podStartSLOduration=2.563497904 podStartE2EDuration="2.563497904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:02.561207066 +0000 UTC m=+1114.478228639" watchObservedRunningTime="2026-10-30 20:45:02.563497904 +0000 UTC m=+1114.480519439"
Oct 30 20:45:02.676553 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:02.676487    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889885-5lxxx" podStartSLOduration=2.676469395 podStartE2EDuration="2.676469395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:02.67563078 +0000 UTC m=+1114.592652339" watchObservedRunningTime="2026-10-30 20:45:02.676469395 +0000 UTC m=+1114.593490940"
Oct 30 20:45:03.087502 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:03.087460    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f6ebf5-29889885-k6svg" event={"ID":"3892545f-5bd2-490b-bd69-b1e8434da80c","Type":"ContainerStarted","Data":"3c18c87f0cd41743442bfdcf7d8abe475ee8a98dc9d5aa7a814d700f2e99c6fd"}
Oct 30 20:45:03.089215 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:03.089186    5071 generic.go:356] "Generic (PLEG): container finished" podID="26471d92-815c-471c-b8be-52f4fe79dbc8" containerID="6a975508fae807349c96461b66baf3b33ee8c7bf0a67cf843d88f79451a7e05d" exitCode=60
Oct 30 20:45:03.089348 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:03.089266    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889885-5lxxx" event={"ID":"26471d92-815c-471c-b8be-52f4fe79dbc8","Type":"ContainerDied","Data":"6a975508fae807349c96461b66baf3b33ee8c7bf0a67cf843d88f79451a7e05d"}
Oct 30 20:45:03.089533 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:03.089511    5071 scope.go:122] "RemoveContainer" containerID="6a975508fae807349c96461b66baf3b33ee8c7bf0a67cf843d88f79451a7e05d"
Oct 30 20:45:03.091172 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:03.091091    5071 generic.go:356] "Generic (PLEG): container finished" podID="4ee26fb5-c5d7-4b70-a994-67b1c487e5da" containerID="c178d91d805867b1f5206cb8611a197aa4e43bfa5b655eca41df61d48834dfea" exitCode=60
Oct 30 20:45:03.091504 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:03.091483    5071 scope.go:122] "RemoveContainer" containerID="c178d91d805867b1f5206cb8611a197aa4e43bfa5b655eca41df61d48834dfea"
Oct 30 20:45:03.091808 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:03.091785    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-6nmnq" event={"ID":"4ee26fb5-c5d7-4b70-a994-67b1c487e5da","Type":"ContainerDied","Data":"c178d91d805867b1f5206cb8611a197aa4e43bfa5b655eca41df61d48834dfea"}
Oct 30 20:45:03.094601 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:03.094575    5071 generic.go:356] "Generic (PLEG): container finished" podID="0729ed1c-bbfd-48a8-97aa-f2fe75735a9e" containerID="cc7470c9993b97ebd555c5a326f03cf4dea437c25c095144dc4856e6e575ab34" exitCode=60
Oct 30 20:45:03.094849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:03.094797    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889885-cvfkd" event={"ID":"0729ed1c-bbfd-48a8-97aa-f2fe75735a9e","Type":"ContainerDied","Data":"cc7470c9993b97ebd555c5a326f03cf4dea437c25c095144dc4856e6e575ab34"}
Oct 30 20:45:03.095294 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:03.095275    5071 scope.go:122] "RemoveContainer" containerID="cc7470c9993b97ebd555c5a326f03cf4dea437c25c095144dc4856e6e575ab34"
Oct 30 20:45:03.210730 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:03.210685    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f6ebf5-29889885-k6svg" podStartSLOduration=3.210663526 podStartE2EDuration="3.210663526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:03.206896498 +0000 UTC m=+1115.123918034" watchObservedRunningTime="2026-10-30 20:45:03.210663526 +0000 UTC m=+1115.127685081"
Oct 30 20:45:03.306774 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:03.306735    5071 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 20:45:03.307168 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:03.306787    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz" podUID="203f757a-7115-466c-8a13-14fb326599ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 20:45:04.104176 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:04.104135    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889885-cvfkd" event={"ID":"0729ed1c-bbfd-48a8-97aa-f2fe75735a9e","Type":"ContainerStarted","Data":"0e19c41580290cbeaf987de4db8ed8caa98d4654a9d68333eafc84033bcac234"}
Oct 30 20:45:04.105998 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:04.105967    5071 generic.go:356] "Generic (PLEG): container finished" podID="3892545f-5bd2-490b-bd69-b1e8434da80c" containerID="3c18c87f0cd41743442bfdcf7d8abe475ee8a98dc9d5aa7a814d700f2e99c6fd" exitCode=60
Oct 30 20:45:04.106150 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:04.106030    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f6ebf5-29889885-k6svg" event={"ID":"3892545f-5bd2-490b-bd69-b1e8434da80c","Type":"ContainerDied","Data":"3c18c87f0cd41743442bfdcf7d8abe475ee8a98dc9d5aa7a814d700f2e99c6fd"}
Oct 30 20:45:04.106150 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:04.106072    5071 scope.go:122] "RemoveContainer" containerID="527eccfd8d6d9fb9fec90883c366b626a83290a3514604a7b9d3487d14ccd86d"
Oct 30 20:45:04.106312 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:04.106294    5071 scope.go:122] "RemoveContainer" containerID="3c18c87f0cd41743442bfdcf7d8abe475ee8a98dc9d5aa7a814d700f2e99c6fd"
Oct 30 20:45:04.106617 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:45:04.106583    5071 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f6ebf5-29889885-k6svg_odh-ai-gateway-infra(3892545f-5bd2-490b-bd69-b1e8434da80c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f6ebf5-29889885-k6svg" podUID="3892545f-5bd2-490b-bd69-b1e8434da80c"
Oct 30 20:45:04.108235 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:04.108211    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889885-5lxxx" event={"ID":"26471d92-815c-471c-b8be-52f4fe79dbc8","Type":"ContainerStarted","Data":"fb74345f409615d97673d3e098f749f46abbcae782f2153985df5f3293350808"}
Oct 30 20:45:04.110005 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:04.109984    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-6nmnq" event={"ID":"4ee26fb5-c5d7-4b70-a994-67b1c487e5da","Type":"ContainerStarted","Data":"2ec5764e0ebacf8fab2af53d09b1abc12019aa0ac30faa4440fc2cdb95e548e5"}
Oct 30 20:45:04.306250 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:04.306223    5071 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 20:45:04.306365 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:04.306275    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz" podUID="203f757a-7115-466c-8a13-14fb326599ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 20:45:05.119184 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:05.119143    5071 generic.go:356] "Generic (PLEG): container finished" podID="0729ed1c-bbfd-48a8-97aa-f2fe75735a9e" containerID="0e19c41580290cbeaf987de4db8ed8caa98d4654a9d68333eafc84033bcac234" exitCode=60
Oct 30 20:45:05.119703 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:05.119234    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889885-cvfkd" event={"ID":"0729ed1c-bbfd-48a8-97aa-f2fe75735a9e","Type":"ContainerDied","Data":"0e19c41580290cbeaf987de4db8ed8caa98d4654a9d68333eafc84033bcac234"}
Oct 30 20:45:05.119703 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:05.119308    5071 scope.go:122] "RemoveContainer" containerID="cc7470c9993b97ebd555c5a326f03cf4dea437c25c095144dc4856e6e575ab34"
Oct 30 20:45:05.119703 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:05.119537    5071 scope.go:122] "RemoveContainer" containerID="0e19c41580290cbeaf987de4db8ed8caa98d4654a9d68333eafc84033bcac234"
Oct 30 20:45:05.119929 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:45:05.119909    5071 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889885-cvfkd_odh-ai-gateway-infra(0729ed1c-bbfd-48a8-97aa-f2fe75735a9e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889885-cvfkd" podUID="0729ed1c-bbfd-48a8-97aa-f2fe75735a9e"
Oct 30 20:45:05.121506 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:05.121488    5071 scope.go:122] "RemoveContainer" containerID="3c18c87f0cd41743442bfdcf7d8abe475ee8a98dc9d5aa7a814d700f2e99c6fd"
Oct 30 20:45:05.121805 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:45:05.121785    5071 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f6ebf5-29889885-k6svg_odh-ai-gateway-infra(3892545f-5bd2-490b-bd69-b1e8434da80c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f6ebf5-29889885-k6svg" podUID="3892545f-5bd2-490b-bd69-b1e8434da80c"
Oct 30 20:45:05.123067 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:05.123046    5071 generic.go:356] "Generic (PLEG): container finished" podID="26471d92-815c-471c-b8be-52f4fe79dbc8" containerID="fb74345f409615d97673d3e098f749f46abbcae782f2153985df5f3293350808" exitCode=60
Oct 30 20:45:05.123146 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:05.123125    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889885-5lxxx" event={"ID":"26471d92-815c-471c-b8be-52f4fe79dbc8","Type":"ContainerDied","Data":"fb74345f409615d97673d3e098f749f46abbcae782f2153985df5f3293350808"}
Oct 30 20:45:05.123304 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:05.123289    5071 scope.go:122] "RemoveContainer" containerID="fb74345f409615d97673d3e098f749f46abbcae782f2153985df5f3293350808"
Oct 30 20:45:05.123512 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:45:05.123491    5071 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-386743-29889885-5lxxx_odh-ai-gateway-infra(26471d92-815c-471c-b8be-52f4fe79dbc8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889885-5lxxx" podUID="26471d92-815c-471c-b8be-52f4fe79dbc8"
Oct 30 20:45:05.125226 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:05.125203    5071 generic.go:356] "Generic (PLEG): container finished" podID="4ee26fb5-c5d7-4b70-a994-67b1c487e5da" containerID="2ec5764e0ebacf8fab2af53d09b1abc12019aa0ac30faa4440fc2cdb95e548e5" exitCode=60
Oct 30 20:45:05.125329 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:05.125264    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-6nmnq" event={"ID":"4ee26fb5-c5d7-4b70-a994-67b1c487e5da","Type":"ContainerDied","Data":"2ec5764e0ebacf8fab2af53d09b1abc12019aa0ac30faa4440fc2cdb95e548e5"}
Oct 30 20:45:05.125423 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:05.125409    5071 scope.go:122] "RemoveContainer" containerID="2ec5764e0ebacf8fab2af53d09b1abc12019aa0ac30faa4440fc2cdb95e548e5"
Oct 30 20:45:05.125605 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:45:05.125588    5071 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889885-6nmnq_odh-ai-gateway-infra(4ee26fb5-c5d7-4b70-a994-67b1c487e5da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-6nmnq" podUID="4ee26fb5-c5d7-4b70-a994-67b1c487e5da"
Oct 30 20:45:05.156473 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:05.156446    5071 scope.go:122] "RemoveContainer" containerID="6a975508fae807349c96461b66baf3b33ee8c7bf0a67cf843d88f79451a7e05d"
Oct 30 20:45:05.170053 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:05.170028    5071 scope.go:122] "RemoveContainer" containerID="c178d91d805867b1f5206cb8611a197aa4e43bfa5b655eca41df61d48834dfea"
Oct 30 20:45:05.306352 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:05.306319    5071 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 20:45:05.306520 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:05.306378    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz" podUID="203f757a-7115-466c-8a13-14fb326599ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 20:45:06.163848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:06.162927    5071 scope.go:122] "RemoveContainer" containerID="2ec5764e0ebacf8fab2af53d09b1abc12019aa0ac30faa4440fc2cdb95e548e5"
Oct 30 20:45:06.163848 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:45:06.163298    5071 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889885-6nmnq_odh-ai-gateway-infra(4ee26fb5-c5d7-4b70-a994-67b1c487e5da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-6nmnq" podUID="4ee26fb5-c5d7-4b70-a994-67b1c487e5da"
Oct 30 20:45:06.166617 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:06.166586    5071 scope.go:122] "RemoveContainer" containerID="0e19c41580290cbeaf987de4db8ed8caa98d4654a9d68333eafc84033bcac234"
Oct 30 20:45:06.166984 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:45:06.166955    5071 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889885-cvfkd_odh-ai-gateway-infra(0729ed1c-bbfd-48a8-97aa-f2fe75735a9e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889885-cvfkd" podUID="0729ed1c-bbfd-48a8-97aa-f2fe75735a9e"
Oct 30 20:45:06.169863 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:06.168343    5071 scope.go:122] "RemoveContainer" containerID="fb74345f409615d97673d3e098f749f46abbcae782f2153985df5f3293350808"
Oct 30 20:45:06.169863 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:45:06.168579    5071 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-386743-29889885-5lxxx_odh-ai-gateway-infra(26471d92-815c-471c-b8be-52f4fe79dbc8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889885-5lxxx" podUID="26471d92-815c-471c-b8be-52f4fe79dbc8"
Oct 30 20:45:06.306956 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:06.306920    5071 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 20:45:06.307162 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:06.307002    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz" podUID="203f757a-7115-466c-8a13-14fb326599ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 20:45:06.915261 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:06.915231    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:07.306803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:07.306769    5071 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 20:45:07.307312 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:07.306839    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz" podUID="203f757a-7115-466c-8a13-14fb326599ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 20:45:08.220030 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:08.219992    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:08.320698 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:08.320662    5071 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 20:45:08.321225 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:08.320731    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz" podUID="203f757a-7115-466c-8a13-14fb326599ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 20:45:09.311433 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:09.311404    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"
Oct 30 20:45:09.311807 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:09.311785    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"
Oct 30 20:45:09.312499 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:09.312480    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"
Oct 30 20:45:12.008810 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:12.008770    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:15.194120 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:15.194089    5071 scope.go:122] "RemoveContainer" containerID="fb74345f409615d97673d3e098f749f46abbcae782f2153985df5f3293350808"
Oct 30 20:45:16.103870 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.103838    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"]
Oct 30 20:45:16.108179 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.108158    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"
Oct 30 20:45:16.172763 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.172722    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"]
Oct 30 20:45:16.203740 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.203702    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbc434f-60ea-4a24-9115-b5db756cbec2-credential-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq\" (UID: \"3dbc434f-60ea-4a24-9115-b5db756cbec2\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"
Oct 30 20:45:16.204309 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.203926    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbc434f-60ea-4a24-9115-b5db756cbec2-istio-data\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq\" (UID: \"3dbc434f-60ea-4a24-9115-b5db756cbec2\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"
Oct 30 20:45:16.204309 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.204048    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbc434f-60ea-4a24-9115-b5db756cbec2-istio-podinfo\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq\" (UID: \"3dbc434f-60ea-4a24-9115-b5db756cbec2\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"
Oct 30 20:45:16.204309 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.204091    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbc434f-60ea-4a24-9115-b5db756cbec2-workload-certs\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq\" (UID: \"3dbc434f-60ea-4a24-9115-b5db756cbec2\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"
Oct 30 20:45:16.204309 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.204125    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbc434f-60ea-4a24-9115-b5db756cbec2-istio-envoy\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq\" (UID: \"3dbc434f-60ea-4a24-9115-b5db756cbec2\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"
Oct 30 20:45:16.204309 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.204185    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbc434f-60ea-4a24-9115-b5db756cbec2-workload-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq\" (UID: \"3dbc434f-60ea-4a24-9115-b5db756cbec2\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"
Oct 30 20:45:16.204309 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.204275    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbc434f-60ea-4a24-9115-b5db756cbec2-istiod-ca-cert\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq\" (UID: \"3dbc434f-60ea-4a24-9115-b5db756cbec2\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"
Oct 30 20:45:16.204627 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.204337    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf2vt\" (UniqueName: \"kubernetes.io/projected/3dbc434f-60ea-4a24-9115-b5db756cbec2-kube-api-access-zf2vt\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq\" (UID: \"3dbc434f-60ea-4a24-9115-b5db756cbec2\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"
Oct 30 20:45:16.204627 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.204420    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dbc434f-60ea-4a24-9115-b5db756cbec2-istio-token\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq\" (UID: \"3dbc434f-60ea-4a24-9115-b5db756cbec2\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"
Oct 30 20:45:16.211016 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.210992    5071 scope.go:122] "RemoveContainer" containerID="3c18c87f0cd41743442bfdcf7d8abe475ee8a98dc9d5aa7a814d700f2e99c6fd"
Oct 30 20:45:16.261571 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.261534    5071 generic.go:356] "Generic (PLEG): container finished" podID="26471d92-815c-471c-b8be-52f4fe79dbc8" containerID="acf65dd1891fd4179e6ffd009597f457af017bc9fc7a3812b83db5cc4657bb26" exitCode=60
Oct 30 20:45:16.261703 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.261600    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889885-5lxxx" event={"ID":"26471d92-815c-471c-b8be-52f4fe79dbc8","Type":"ContainerDied","Data":"acf65dd1891fd4179e6ffd009597f457af017bc9fc7a3812b83db5cc4657bb26"}
Oct 30 20:45:16.261703 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.261662    5071 scope.go:122] "RemoveContainer" containerID="fb74345f409615d97673d3e098f749f46abbcae782f2153985df5f3293350808"
Oct 30 20:45:16.262044 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.262028    5071 scope.go:122] "RemoveContainer" containerID="acf65dd1891fd4179e6ffd009597f457af017bc9fc7a3812b83db5cc4657bb26"
Oct 30 20:45:16.262302 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:45:16.262283    5071 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-386743-29889885-5lxxx_odh-ai-gateway-infra(26471d92-815c-471c-b8be-52f4fe79dbc8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889885-5lxxx" podUID="26471d92-815c-471c-b8be-52f4fe79dbc8"
Oct 30 20:45:16.305537 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.305513    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbc434f-60ea-4a24-9115-b5db756cbec2-istiod-ca-cert\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq\" (UID: \"3dbc434f-60ea-4a24-9115-b5db756cbec2\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"
Oct 30 20:45:16.305674 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.305580    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf2vt\" (UniqueName: \"kubernetes.io/projected/3dbc434f-60ea-4a24-9115-b5db756cbec2-kube-api-access-zf2vt\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq\" (UID: \"3dbc434f-60ea-4a24-9115-b5db756cbec2\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"
Oct 30 20:45:16.305674 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.305636    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dbc434f-60ea-4a24-9115-b5db756cbec2-istio-token\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq\" (UID: \"3dbc434f-60ea-4a24-9115-b5db756cbec2\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"
Oct 30 20:45:16.305800 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.305741    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbc434f-60ea-4a24-9115-b5db756cbec2-credential-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq\" (UID: \"3dbc434f-60ea-4a24-9115-b5db756cbec2\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"
Oct 30 20:45:16.305984 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.305965    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbc434f-60ea-4a24-9115-b5db756cbec2-istio-data\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq\" (UID: \"3dbc434f-60ea-4a24-9115-b5db756cbec2\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"
Oct 30 20:45:16.306099 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.306061    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbc434f-60ea-4a24-9115-b5db756cbec2-istio-podinfo\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq\" (UID: \"3dbc434f-60ea-4a24-9115-b5db756cbec2\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"
Oct 30 20:45:16.306200 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.306127    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbc434f-60ea-4a24-9115-b5db756cbec2-workload-certs\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq\" (UID: \"3dbc434f-60ea-4a24-9115-b5db756cbec2\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"
Oct 30 20:45:16.306200 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.306153    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbc434f-60ea-4a24-9115-b5db756cbec2-credential-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq\" (UID: \"3dbc434f-60ea-4a24-9115-b5db756cbec2\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"
Oct 30 20:45:16.306200 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.306164    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbc434f-60ea-4a24-9115-b5db756cbec2-istio-envoy\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq\" (UID: \"3dbc434f-60ea-4a24-9115-b5db756cbec2\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"
Oct 30 20:45:16.306436 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.306223    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbc434f-60ea-4a24-9115-b5db756cbec2-workload-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq\" (UID: \"3dbc434f-60ea-4a24-9115-b5db756cbec2\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"
Oct 30 20:45:16.306436 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.306259    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbc434f-60ea-4a24-9115-b5db756cbec2-istiod-ca-cert\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq\" (UID: \"3dbc434f-60ea-4a24-9115-b5db756cbec2\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"
Oct 30 20:45:16.306607 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.306585    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbc434f-60ea-4a24-9115-b5db756cbec2-workload-certs\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq\" (UID: \"3dbc434f-60ea-4a24-9115-b5db756cbec2\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"
Oct 30 20:45:16.306673 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.306613    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbc434f-60ea-4a24-9115-b5db756cbec2-istio-data\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq\" (UID: \"3dbc434f-60ea-4a24-9115-b5db756cbec2\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"
Oct 30 20:45:16.306973 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.306810    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbc434f-60ea-4a24-9115-b5db756cbec2-workload-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq\" (UID: \"3dbc434f-60ea-4a24-9115-b5db756cbec2\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"
Oct 30 20:45:16.308518 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.308499    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbc434f-60ea-4a24-9115-b5db756cbec2-istio-envoy\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq\" (UID: \"3dbc434f-60ea-4a24-9115-b5db756cbec2\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"
Oct 30 20:45:16.308845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.308806    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbc434f-60ea-4a24-9115-b5db756cbec2-istio-podinfo\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq\" (UID: \"3dbc434f-60ea-4a24-9115-b5db756cbec2\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"
Oct 30 20:45:16.364362 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.364331    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dbc434f-60ea-4a24-9115-b5db756cbec2-istio-token\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq\" (UID: \"3dbc434f-60ea-4a24-9115-b5db756cbec2\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"
Oct 30 20:45:16.364518 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.364498    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf2vt\" (UniqueName: \"kubernetes.io/projected/3dbc434f-60ea-4a24-9115-b5db756cbec2-kube-api-access-zf2vt\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq\" (UID: \"3dbc434f-60ea-4a24-9115-b5db756cbec2\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"
Oct 30 20:45:16.422644 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.422613    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"
Oct 30 20:45:16.735059 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:45:16.735020    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:45:16.857461 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.857435    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"]
Oct 30 20:45:16.857695 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:45:16.857653    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dbc434f_60ea_4a24_9115_b5db756cbec2.slice/crio-adc6a78935070e28c34685e75122f7f195f9d4e962b0adcce6949ca7c1bb5414 WatchSource:0}: Error finding container adc6a78935070e28c34685e75122f7f195f9d4e962b0adcce6949ca7c1bb5414: Status 404 returned error can't find the container with id adc6a78935070e28c34685e75122f7f195f9d4e962b0adcce6949ca7c1bb5414
Oct 30 20:45:16.860516 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.860481    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:45:16.860591 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.860567    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:45:16.860631 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.860613    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:45:16.899752 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:16.899713    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:17.194808 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:17.194731    5071 scope.go:122] "RemoveContainer" containerID="2ec5764e0ebacf8fab2af53d09b1abc12019aa0ac30faa4440fc2cdb95e548e5"
Oct 30 20:45:17.277156 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:17.277122    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq" event={"ID":"3dbc434f-60ea-4a24-9115-b5db756cbec2","Type":"ContainerStarted","Data":"969c705a753fac45f41fb6ae2a8c853b62afd3c595a31498994a0428241d913f"}
Oct 30 20:45:17.277591 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:17.277180    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq" event={"ID":"3dbc434f-60ea-4a24-9115-b5db756cbec2","Type":"ContainerStarted","Data":"adc6a78935070e28c34685e75122f7f195f9d4e962b0adcce6949ca7c1bb5414"}
Oct 30 20:45:17.279387 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:17.279354    5071 generic.go:356] "Generic (PLEG): container finished" podID="3892545f-5bd2-490b-bd69-b1e8434da80c" containerID="4950a7dfec6e033aa1fd82c645854f6d5dbcd9a9dd634eb67e90968e139786fe" exitCode=60
Oct 30 20:45:17.279510 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:17.279422    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f6ebf5-29889885-k6svg" event={"ID":"3892545f-5bd2-490b-bd69-b1e8434da80c","Type":"ContainerDied","Data":"4950a7dfec6e033aa1fd82c645854f6d5dbcd9a9dd634eb67e90968e139786fe"}
Oct 30 20:45:17.279510 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:17.279469    5071 scope.go:122] "RemoveContainer" containerID="3c18c87f0cd41743442bfdcf7d8abe475ee8a98dc9d5aa7a814d700f2e99c6fd"
Oct 30 20:45:17.279866 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:17.279813    5071 scope.go:122] "RemoveContainer" containerID="4950a7dfec6e033aa1fd82c645854f6d5dbcd9a9dd634eb67e90968e139786fe"
Oct 30 20:45:17.280182 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:45:17.280133    5071 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f6ebf5-29889885-k6svg_odh-ai-gateway-infra(3892545f-5bd2-490b-bd69-b1e8434da80c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f6ebf5-29889885-k6svg" podUID="3892545f-5bd2-490b-bd69-b1e8434da80c"
Oct 30 20:45:17.390675 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:17.390636    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889885-5lxxx"]
Oct 30 20:45:17.401754 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:17.401705    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq" podStartSLOduration=1.4016881460000001 podStartE2EDuration="1.401688146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:17.398216086 +0000 UTC m=+1129.315237621" watchObservedRunningTime="2026-10-30 20:45:17.401688146 +0000 UTC m=+1129.318709682"
Oct 30 20:45:17.423896 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:17.423815    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"
Oct 30 20:45:17.427867 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:17.425745    5071 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 20:45:17.427867 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:17.425806    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq" podUID="3dbc434f-60ea-4a24-9115-b5db756cbec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 20:45:17.659009 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:17.658619    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889885-5lxxx"
Oct 30 20:45:17.757517 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:17.757417    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26471d92-815c-471c-b8be-52f4fe79dbc8-kube-api-access-q7g59\" (UniqueName: \"kubernetes.io/projected/26471d92-815c-471c-b8be-52f4fe79dbc8-kube-api-access-q7g59\") pod \"26471d92-815c-471c-b8be-52f4fe79dbc8\" (UID: \"26471d92-815c-471c-b8be-52f4fe79dbc8\") "
Oct 30 20:45:17.757677 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:17.757615    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26471d92-815c-471c-b8be-52f4fe79dbc8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26471d92-815c-471c-b8be-52f4fe79dbc8-maas-api-service-ca\") pod \"26471d92-815c-471c-b8be-52f4fe79dbc8\" (UID: \"26471d92-815c-471c-b8be-52f4fe79dbc8\") "
Oct 30 20:45:17.758043 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:17.758019    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26471d92-815c-471c-b8be-52f4fe79dbc8-maas-api-service-ca" pod "26471d92-815c-471c-b8be-52f4fe79dbc8" (UID: "26471d92-815c-471c-b8be-52f4fe79dbc8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:45:17.758163 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:17.758148    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26471d92-815c-471c-b8be-52f4fe79dbc8-maas-api-service-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:17.759803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:17.759778    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26471d92-815c-471c-b8be-52f4fe79dbc8-kube-api-access-q7g59" pod "26471d92-815c-471c-b8be-52f4fe79dbc8" (UID: "26471d92-815c-471c-b8be-52f4fe79dbc8"). InnerVolumeSpecName "kube-api-access-q7g59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:45:17.858999 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:17.858968    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7g59\" (UniqueName: \"kubernetes.io/projected/26471d92-815c-471c-b8be-52f4fe79dbc8-kube-api-access-q7g59\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:18.199439 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:18.199403    5071 scope.go:122] "RemoveContainer" containerID="0e19c41580290cbeaf987de4db8ed8caa98d4654a9d68333eafc84033bcac234"
Oct 30 20:45:18.290615 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:18.290581    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-6nmnq" event={"ID":"4ee26fb5-c5d7-4b70-a994-67b1c487e5da","Type":"ContainerStarted","Data":"01d2c8c5a3b3bd4374af010570884b814c34d733aa2af166a03ace5ff0eeffac"}
Oct 30 20:45:18.293944 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:18.293920    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889885-5lxxx"
Oct 30 20:45:18.294101 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:18.293951    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889885-5lxxx" event={"ID":"26471d92-815c-471c-b8be-52f4fe79dbc8","Type":"ContainerDied","Data":"d9a8ba11734ea15547b260006522457deeb0562926b41e6f7b05629fae097fe6"}
Oct 30 20:45:18.294101 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:18.294008    5071 scope.go:122] "RemoveContainer" containerID="acf65dd1891fd4179e6ffd009597f457af017bc9fc7a3812b83db5cc4657bb26"
Oct 30 20:45:18.423041 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:18.423009    5071 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 20:45:18.423307 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:18.423275    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq" podUID="3dbc434f-60ea-4a24-9115-b5db756cbec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 20:45:18.549089 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:18.548020    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889885-5lxxx"]
Oct 30 20:45:18.561492 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:18.561390    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889885-5lxxx"]
Oct 30 20:45:18.666300 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:18.666257    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f6ebf5-29889885-k6svg"]
Oct 30 20:45:18.902098 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:18.902069    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f6ebf5-29889885-k6svg"
Oct 30 20:45:19.074650 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:19.073587    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3892545f-5bd2-490b-bd69-b1e8434da80c-kube-api-access-rxx4b\" (UniqueName: \"kubernetes.io/projected/3892545f-5bd2-490b-bd69-b1e8434da80c-kube-api-access-rxx4b\") pod \"3892545f-5bd2-490b-bd69-b1e8434da80c\" (UID: \"3892545f-5bd2-490b-bd69-b1e8434da80c\") "
Oct 30 20:45:19.074650 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:19.073655    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3892545f-5bd2-490b-bd69-b1e8434da80c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3892545f-5bd2-490b-bd69-b1e8434da80c-maas-api-service-ca\") pod \"3892545f-5bd2-490b-bd69-b1e8434da80c\" (UID: \"3892545f-5bd2-490b-bd69-b1e8434da80c\") "
Oct 30 20:45:19.074650 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:19.074609    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3892545f-5bd2-490b-bd69-b1e8434da80c-maas-api-service-ca" pod "3892545f-5bd2-490b-bd69-b1e8434da80c" (UID: "3892545f-5bd2-490b-bd69-b1e8434da80c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:45:19.077112 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:19.077082    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3892545f-5bd2-490b-bd69-b1e8434da80c-kube-api-access-rxx4b" pod "3892545f-5bd2-490b-bd69-b1e8434da80c" (UID: "3892545f-5bd2-490b-bd69-b1e8434da80c"). InnerVolumeSpecName "kube-api-access-rxx4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:45:19.175866 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:19.175762    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxx4b\" (UniqueName: \"kubernetes.io/projected/3892545f-5bd2-490b-bd69-b1e8434da80c-kube-api-access-rxx4b\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:19.175866 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:19.175794    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3892545f-5bd2-490b-bd69-b1e8434da80c-maas-api-service-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:19.360844 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:19.355580    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:19.360844 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:19.359390    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889885-cvfkd" event={"ID":"0729ed1c-bbfd-48a8-97aa-f2fe75735a9e","Type":"ContainerStarted","Data":"1e15f7fdc5af97981549a93d29b46999ad54696dd456e4039927c82a1d894adc"}
Oct 30 20:45:19.370052 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:19.367556    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f6ebf5-29889885-k6svg" event={"ID":"3892545f-5bd2-490b-bd69-b1e8434da80c","Type":"ContainerDied","Data":"a31246f96ccaa9aab31c4098fb04f8a6a52159162927d41a6f2ffd8979ccfdd0"}
Oct 30 20:45:19.370052 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:19.367633    5071 scope.go:122] "RemoveContainer" containerID="4950a7dfec6e033aa1fd82c645854f6d5dbcd9a9dd634eb67e90968e139786fe"
Oct 30 20:45:19.371672 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:19.371223    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f6ebf5-29889885-k6svg"
Oct 30 20:45:19.470940 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:19.430749    5071 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 20:45:19.470940 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:19.430798    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq" podUID="3dbc434f-60ea-4a24-9115-b5db756cbec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 20:45:19.470940 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:19.434238    5071 generic.go:356] "Generic (PLEG): container finished" podID="4ee26fb5-c5d7-4b70-a994-67b1c487e5da" containerID="01d2c8c5a3b3bd4374af010570884b814c34d733aa2af166a03ace5ff0eeffac" exitCode=60
Oct 30 20:45:19.470940 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:19.435206    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-6nmnq" event={"ID":"4ee26fb5-c5d7-4b70-a994-67b1c487e5da","Type":"ContainerDied","Data":"01d2c8c5a3b3bd4374af010570884b814c34d733aa2af166a03ace5ff0eeffac"}
Oct 30 20:45:19.470940 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:19.435565    5071 scope.go:122] "RemoveContainer" containerID="01d2c8c5a3b3bd4374af010570884b814c34d733aa2af166a03ace5ff0eeffac"
Oct 30 20:45:19.470940 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:45:19.435875    5071 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889885-6nmnq_odh-ai-gateway-infra(4ee26fb5-c5d7-4b70-a994-67b1c487e5da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-6nmnq" podUID="4ee26fb5-c5d7-4b70-a994-67b1c487e5da"
Oct 30 20:45:19.481029 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:19.480999    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889885-6nmnq"]
Oct 30 20:45:19.518736 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:19.518707    5071 scope.go:122] "RemoveContainer" containerID="2ec5764e0ebacf8fab2af53d09b1abc12019aa0ac30faa4440fc2cdb95e548e5"
Oct 30 20:45:19.681884 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:19.652086    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f6ebf5-29889885-k6svg"]
Oct 30 20:45:19.691865 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:19.691813    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f6ebf5-29889885-k6svg"]
Oct 30 20:45:20.202508 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:20.202475    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26471d92-815c-471c-b8be-52f4fe79dbc8" path="/var/lib/kubelet/pods/26471d92-815c-471c-b8be-52f4fe79dbc8/volumes"
Oct 30 20:45:20.202936 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:20.202922    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3892545f-5bd2-490b-bd69-b1e8434da80c" path="/var/lib/kubelet/pods/3892545f-5bd2-490b-bd69-b1e8434da80c/volumes"
Oct 30 20:45:20.423698 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:20.423662    5071 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 20:45:20.424112 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:20.423718    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq" podUID="3dbc434f-60ea-4a24-9115-b5db756cbec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 20:45:20.468647 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:20.468615    5071 generic.go:356] "Generic (PLEG): container finished" podID="0729ed1c-bbfd-48a8-97aa-f2fe75735a9e" containerID="1e15f7fdc5af97981549a93d29b46999ad54696dd456e4039927c82a1d894adc" exitCode=60
Oct 30 20:45:20.468959 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:20.468936    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889885-cvfkd" event={"ID":"0729ed1c-bbfd-48a8-97aa-f2fe75735a9e","Type":"ContainerDied","Data":"1e15f7fdc5af97981549a93d29b46999ad54696dd456e4039927c82a1d894adc"}
Oct 30 20:45:20.469113 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:20.469097    5071 scope.go:122] "RemoveContainer" containerID="0e19c41580290cbeaf987de4db8ed8caa98d4654a9d68333eafc84033bcac234"
Oct 30 20:45:20.469658 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:20.469642    5071 scope.go:122] "RemoveContainer" containerID="1e15f7fdc5af97981549a93d29b46999ad54696dd456e4039927c82a1d894adc"
Oct 30 20:45:20.470193 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:45:20.470163    5071 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889885-cvfkd_odh-ai-gateway-infra(0729ed1c-bbfd-48a8-97aa-f2fe75735a9e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889885-cvfkd" podUID="0729ed1c-bbfd-48a8-97aa-f2fe75735a9e"
Oct 30 20:45:20.512856 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:20.512120    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889885-cvfkd"]
Oct 30 20:45:20.704358 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:20.704337    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-6nmnq"
Oct 30 20:45:20.807173 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:20.807138    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ee26fb5-c5d7-4b70-a994-67b1c487e5da-kube-api-access-t2dv9\" (UniqueName: \"kubernetes.io/projected/4ee26fb5-c5d7-4b70-a994-67b1c487e5da-kube-api-access-t2dv9\") pod \"4ee26fb5-c5d7-4b70-a994-67b1c487e5da\" (UID: \"4ee26fb5-c5d7-4b70-a994-67b1c487e5da\") "
Oct 30 20:45:20.807438 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:20.807421    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ee26fb5-c5d7-4b70-a994-67b1c487e5da-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ee26fb5-c5d7-4b70-a994-67b1c487e5da-maas-api-service-ca\") pod \"4ee26fb5-c5d7-4b70-a994-67b1c487e5da\" (UID: \"4ee26fb5-c5d7-4b70-a994-67b1c487e5da\") "
Oct 30 20:45:20.807856 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:20.807811    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ee26fb5-c5d7-4b70-a994-67b1c487e5da-maas-api-service-ca" pod "4ee26fb5-c5d7-4b70-a994-67b1c487e5da" (UID: "4ee26fb5-c5d7-4b70-a994-67b1c487e5da"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:45:20.809506 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:20.809477    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ee26fb5-c5d7-4b70-a994-67b1c487e5da-kube-api-access-t2dv9" pod "4ee26fb5-c5d7-4b70-a994-67b1c487e5da" (UID: "4ee26fb5-c5d7-4b70-a994-67b1c487e5da"). InnerVolumeSpecName "kube-api-access-t2dv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:45:20.909330 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:20.909253    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2dv9\" (UniqueName: \"kubernetes.io/projected/4ee26fb5-c5d7-4b70-a994-67b1c487e5da-kube-api-access-t2dv9\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:20.909330 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:20.909281    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ee26fb5-c5d7-4b70-a994-67b1c487e5da-maas-api-service-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:21.423994 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:21.423957    5071 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 20:45:21.424402 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:21.424040    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq" podUID="3dbc434f-60ea-4a24-9115-b5db756cbec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 20:45:21.487785 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:21.487752    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-6nmnq" event={"ID":"4ee26fb5-c5d7-4b70-a994-67b1c487e5da","Type":"ContainerDied","Data":"ff6f093390b39bf4112806264a259dd288d528aee79a51732d80b0cbe400a46c"}
Oct 30 20:45:21.487946 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:21.487805    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-6nmnq"
Oct 30 20:45:21.487946 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:21.487855    5071 scope.go:122] "RemoveContainer" containerID="01d2c8c5a3b3bd4374af010570884b814c34d733aa2af166a03ace5ff0eeffac"
Oct 30 20:45:21.689340 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:21.689277    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889885-cvfkd"
Oct 30 20:45:21.753385 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:21.753347    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889885-6nmnq"]
Oct 30 20:45:21.755576 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:21.755545    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0729ed1c-bbfd-48a8-97aa-f2fe75735a9e-kube-api-access-spgn2\" (UniqueName: \"kubernetes.io/projected/0729ed1c-bbfd-48a8-97aa-f2fe75735a9e-kube-api-access-spgn2\") pod \"0729ed1c-bbfd-48a8-97aa-f2fe75735a9e\" (UID: \"0729ed1c-bbfd-48a8-97aa-f2fe75735a9e\") "
Oct 30 20:45:21.755716 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:21.755603    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0729ed1c-bbfd-48a8-97aa-f2fe75735a9e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0729ed1c-bbfd-48a8-97aa-f2fe75735a9e-maas-api-service-ca\") pod \"0729ed1c-bbfd-48a8-97aa-f2fe75735a9e\" (UID: \"0729ed1c-bbfd-48a8-97aa-f2fe75735a9e\") "
Oct 30 20:45:21.756005 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:21.755973    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0729ed1c-bbfd-48a8-97aa-f2fe75735a9e-maas-api-service-ca" pod "0729ed1c-bbfd-48a8-97aa-f2fe75735a9e" (UID: "0729ed1c-bbfd-48a8-97aa-f2fe75735a9e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:45:21.756448 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:21.756393    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0729ed1c-bbfd-48a8-97aa-f2fe75735a9e-maas-api-service-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:21.757633 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:21.757607    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889885-6nmnq"]
Oct 30 20:45:21.758004 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:21.757987    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0729ed1c-bbfd-48a8-97aa-f2fe75735a9e-kube-api-access-spgn2" pod "0729ed1c-bbfd-48a8-97aa-f2fe75735a9e" (UID: "0729ed1c-bbfd-48a8-97aa-f2fe75735a9e"). InnerVolumeSpecName "kube-api-access-spgn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:45:21.858046 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:21.858010    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spgn2\" (UniqueName: \"kubernetes.io/projected/0729ed1c-bbfd-48a8-97aa-f2fe75735a9e-kube-api-access-spgn2\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:22.136631 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:22.136574    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:22.215696 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:22.215654    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ee26fb5-c5d7-4b70-a994-67b1c487e5da" path="/var/lib/kubelet/pods/4ee26fb5-c5d7-4b70-a994-67b1c487e5da/volumes"
Oct 30 20:45:22.424099 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:22.424013    5071 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 20:45:22.424099 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:22.424079    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq" podUID="3dbc434f-60ea-4a24-9115-b5db756cbec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 20:45:22.496117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:22.496079    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889885-cvfkd" event={"ID":"0729ed1c-bbfd-48a8-97aa-f2fe75735a9e","Type":"ContainerDied","Data":"d0066ce8ecb0301daa1a8f6433756f6280f53383c10bf84dd793886b09b4ca88"}
Oct 30 20:45:22.496282 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:22.496157    5071 scope.go:122] "RemoveContainer" containerID="1e15f7fdc5af97981549a93d29b46999ad54696dd456e4039927c82a1d894adc"
Oct 30 20:45:22.496282 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:22.496096    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889885-cvfkd"
Oct 30 20:45:22.665750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:22.665711    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889885-cvfkd"]
Oct 30 20:45:22.684688 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:22.684581    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ee3fad-29889885-cvfkd"]
Oct 30 20:45:23.423199 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:23.423156    5071 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 20:45:23.423412 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:23.423211    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq" podUID="3dbc434f-60ea-4a24-9115-b5db756cbec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 20:45:24.203681 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:24.203630    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0729ed1c-bbfd-48a8-97aa-f2fe75735a9e" path="/var/lib/kubelet/pods/0729ed1c-bbfd-48a8-97aa-f2fe75735a9e/volumes"
Oct 30 20:45:24.427021 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:24.426984    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"
Oct 30 20:45:24.427298 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:24.427275    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"
Oct 30 20:45:24.427897 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:24.427877    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"
Oct 30 20:45:26.734640 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:26.734595    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:27.092859 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:27.092801    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:28.650873 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:28.650747    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:28.808602 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:28.808566    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:30.176510 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:30.176480    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:45:32.227300 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:32.227261    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:33.074476 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:33.074443    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:37.289393 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:37.289360    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:39.797781 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.797748    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31e7414a-ndg6b"]
Oct 30 20:45:39.798393 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.798378    5071 cpu_manager.go:414] "containerMap: removing container" podUID="4ee26fb5-c5d7-4b70-a994-67b1c487e5da" containerName="cleanup"
Oct 30 20:45:39.798437 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.798394    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ee26fb5-c5d7-4b70-a994-67b1c487e5da" containerName="cleanup"
Oct 30 20:45:39.798437 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.798408    5071 cpu_manager.go:414] "containerMap: removing container" podUID="26471d92-815c-471c-b8be-52f4fe79dbc8" containerName="cleanup"
Oct 30 20:45:39.798437 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.798414    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26471d92-815c-471c-b8be-52f4fe79dbc8" containerName="cleanup"
Oct 30 20:45:39.798437 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.798422    5071 cpu_manager.go:414] "containerMap: removing container" podUID="3892545f-5bd2-490b-bd69-b1e8434da80c" containerName="cleanup"
Oct 30 20:45:39.798437 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.798429    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3892545f-5bd2-490b-bd69-b1e8434da80c" containerName="cleanup"
Oct 30 20:45:39.798437 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.798434    5071 cpu_manager.go:414] "containerMap: removing container" podUID="0729ed1c-bbfd-48a8-97aa-f2fe75735a9e" containerName="cleanup"
Oct 30 20:45:39.798437 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.798439    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0729ed1c-bbfd-48a8-97aa-f2fe75735a9e" containerName="cleanup"
Oct 30 20:45:39.798750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.798447    5071 cpu_manager.go:414] "containerMap: removing container" podUID="3892545f-5bd2-490b-bd69-b1e8434da80c" containerName="cleanup"
Oct 30 20:45:39.798750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.798452    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3892545f-5bd2-490b-bd69-b1e8434da80c" containerName="cleanup"
Oct 30 20:45:39.798750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.798458    5071 cpu_manager.go:414] "containerMap: removing container" podUID="26471d92-815c-471c-b8be-52f4fe79dbc8" containerName="cleanup"
Oct 30 20:45:39.798750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.798462    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26471d92-815c-471c-b8be-52f4fe79dbc8" containerName="cleanup"
Oct 30 20:45:39.798750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.798472    5071 cpu_manager.go:414] "containerMap: removing container" podUID="0729ed1c-bbfd-48a8-97aa-f2fe75735a9e" containerName="cleanup"
Oct 30 20:45:39.798750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.798476    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0729ed1c-bbfd-48a8-97aa-f2fe75735a9e" containerName="cleanup"
Oct 30 20:45:39.798750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.798487    5071 cpu_manager.go:414] "containerMap: removing container" podUID="0729ed1c-bbfd-48a8-97aa-f2fe75735a9e" containerName="cleanup"
Oct 30 20:45:39.798750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.798491    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0729ed1c-bbfd-48a8-97aa-f2fe75735a9e" containerName="cleanup"
Oct 30 20:45:39.798750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.798518    5071 cpu_manager.go:414] "containerMap: removing container" podUID="4ee26fb5-c5d7-4b70-a994-67b1c487e5da" containerName="cleanup"
Oct 30 20:45:39.798750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.798522    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ee26fb5-c5d7-4b70-a994-67b1c487e5da" containerName="cleanup"
Oct 30 20:45:39.798750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.798613    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ee26fb5-c5d7-4b70-a994-67b1c487e5da" containerName="cleanup"
Oct 30 20:45:39.798750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.798622    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="3892545f-5bd2-490b-bd69-b1e8434da80c" containerName="cleanup"
Oct 30 20:45:39.798750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.798628    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ee26fb5-c5d7-4b70-a994-67b1c487e5da" containerName="cleanup"
Oct 30 20:45:39.798750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.798635    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="26471d92-815c-471c-b8be-52f4fe79dbc8" containerName="cleanup"
Oct 30 20:45:39.798750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.798640    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="0729ed1c-bbfd-48a8-97aa-f2fe75735a9e" containerName="cleanup"
Oct 30 20:45:39.798750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.798644    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="3892545f-5bd2-490b-bd69-b1e8434da80c" containerName="cleanup"
Oct 30 20:45:39.798750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.798649    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="26471d92-815c-471c-b8be-52f4fe79dbc8" containerName="cleanup"
Oct 30 20:45:39.798750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.798655    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="0729ed1c-bbfd-48a8-97aa-f2fe75735a9e" containerName="cleanup"
Oct 30 20:45:39.803441 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.803420    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e7414a-ndg6b"
Oct 30 20:45:39.844113 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.844074    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32a4c593-1f5c-4060-a418-f216a4bc105f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31e7414a-ndg6b\" (UID: \"32a4c593-1f5c-4060-a418-f216a4bc105f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e7414a-ndg6b"
Oct 30 20:45:39.844306 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.844122    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32a4c593-1f5c-4060-a418-f216a4bc105f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31e7414a-ndg6b\" (UID: \"32a4c593-1f5c-4060-a418-f216a4bc105f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e7414a-ndg6b"
Oct 30 20:45:39.844306 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.844216    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wpqk\" (UniqueName: \"kubernetes.io/projected/32a4c593-1f5c-4060-a418-f216a4bc105f-kube-api-access-4wpqk\") pod \"maas-api-revoke-sub-31e7414a-ndg6b\" (UID: \"32a4c593-1f5c-4060-a418-f216a4bc105f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e7414a-ndg6b"
Oct 30 20:45:39.862065 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.862030    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31e7414a-ndg6b"]
Oct 30 20:45:39.905241 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.905210    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:45:39.905241 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.905241    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 20:45:39.944999 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.944967    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wpqk\" (UniqueName: \"kubernetes.io/projected/32a4c593-1f5c-4060-a418-f216a4bc105f-kube-api-access-4wpqk\") pod \"maas-api-revoke-sub-31e7414a-ndg6b\" (UID: \"32a4c593-1f5c-4060-a418-f216a4bc105f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e7414a-ndg6b"
Oct 30 20:45:39.945195 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.945102    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32a4c593-1f5c-4060-a418-f216a4bc105f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31e7414a-ndg6b\" (UID: \"32a4c593-1f5c-4060-a418-f216a4bc105f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e7414a-ndg6b"
Oct 30 20:45:39.945195 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.945135    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32a4c593-1f5c-4060-a418-f216a4bc105f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31e7414a-ndg6b\" (UID: \"32a4c593-1f5c-4060-a418-f216a4bc105f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e7414a-ndg6b"
Oct 30 20:45:39.945745 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.945722    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32a4c593-1f5c-4060-a418-f216a4bc105f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31e7414a-ndg6b\" (UID: \"32a4c593-1f5c-4060-a418-f216a4bc105f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e7414a-ndg6b"
Oct 30 20:45:39.947601 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:39.947576    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32a4c593-1f5c-4060-a418-f216a4bc105f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31e7414a-ndg6b\" (UID: \"32a4c593-1f5c-4060-a418-f216a4bc105f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e7414a-ndg6b"
Oct 30 20:45:40.003352 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:40.003324    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wpqk\" (UniqueName: \"kubernetes.io/projected/32a4c593-1f5c-4060-a418-f216a4bc105f-kube-api-access-4wpqk\") pod \"maas-api-revoke-sub-31e7414a-ndg6b\" (UID: \"32a4c593-1f5c-4060-a418-f216a4bc105f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e7414a-ndg6b"
Oct 30 20:45:40.121433 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:40.121347    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e7414a-ndg6b"
Oct 30 20:45:40.513460 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:40.513409    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31e7414a-ndg6b"]
Oct 30 20:45:40.516557 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:45:40.516524    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32a4c593_1f5c_4060_a418_f216a4bc105f.slice/crio-cc60e644e79ec0d0c040d8e88299ebab1b3da22ac9da3cef8dda70bb4aab152a WatchSource:0}: Error finding container cc60e644e79ec0d0c040d8e88299ebab1b3da22ac9da3cef8dda70bb4aab152a: Status 404 returned error can't find the container with id cc60e644e79ec0d0c040d8e88299ebab1b3da22ac9da3cef8dda70bb4aab152a
Oct 30 20:45:40.635271 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:40.635230    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e7414a-ndg6b" event={"ID":"32a4c593-1f5c-4060-a418-f216a4bc105f","Type":"ContainerStarted","Data":"cc60e644e79ec0d0c040d8e88299ebab1b3da22ac9da3cef8dda70bb4aab152a"}
Oct 30 20:45:41.644245 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:41.644204    5071 generic.go:356] "Generic (PLEG): container finished" podID="32a4c593-1f5c-4060-a418-f216a4bc105f" containerID="0e4e6c7ce87bc203650b6bdfd85df27f6db80ae784995abdcef7e80885f0a37f" exitCode=0
Oct 30 20:45:41.644630 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:41.644327    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e7414a-ndg6b" event={"ID":"32a4c593-1f5c-4060-a418-f216a4bc105f","Type":"ContainerDied","Data":"0e4e6c7ce87bc203650b6bdfd85df27f6db80ae784995abdcef7e80885f0a37f"}
Oct 30 20:45:42.206286 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:42.206253    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:42.838370 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:42.838338    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e7414a-ndg6b"
Oct 30 20:45:42.878312 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:42.878285    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32a4c593-1f5c-4060-a418-f216a4bc105f-kube-api-access-4wpqk\" (UniqueName: \"kubernetes.io/projected/32a4c593-1f5c-4060-a418-f216a4bc105f-kube-api-access-4wpqk\") pod \"32a4c593-1f5c-4060-a418-f216a4bc105f\" (UID: \"32a4c593-1f5c-4060-a418-f216a4bc105f\") "
Oct 30 20:45:42.878473 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:42.878368    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32a4c593-1f5c-4060-a418-f216a4bc105f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32a4c593-1f5c-4060-a418-f216a4bc105f-maas-api-openshift-ca\") pod \"32a4c593-1f5c-4060-a418-f216a4bc105f\" (UID: \"32a4c593-1f5c-4060-a418-f216a4bc105f\") "
Oct 30 20:45:42.878535 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:42.878477    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32a4c593-1f5c-4060-a418-f216a4bc105f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32a4c593-1f5c-4060-a418-f216a4bc105f-maas-api-xks-ca\") pod \"32a4c593-1f5c-4060-a418-f216a4bc105f\" (UID: \"32a4c593-1f5c-4060-a418-f216a4bc105f\") "
Oct 30 20:45:42.878742 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:42.878715    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32a4c593-1f5c-4060-a418-f216a4bc105f-maas-api-openshift-ca" pod "32a4c593-1f5c-4060-a418-f216a4bc105f" (UID: "32a4c593-1f5c-4060-a418-f216a4bc105f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:45:42.878894 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:42.878877    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32a4c593-1f5c-4060-a418-f216a4bc105f-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:42.880609 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:42.880584    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32a4c593-1f5c-4060-a418-f216a4bc105f-kube-api-access-4wpqk" pod "32a4c593-1f5c-4060-a418-f216a4bc105f" (UID: "32a4c593-1f5c-4060-a418-f216a4bc105f"). InnerVolumeSpecName "kube-api-access-4wpqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:45:42.880755 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:42.880734    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32a4c593-1f5c-4060-a418-f216a4bc105f-maas-api-xks-ca" pod "32a4c593-1f5c-4060-a418-f216a4bc105f" (UID: "32a4c593-1f5c-4060-a418-f216a4bc105f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:45:42.980001 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:42.979922    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wpqk\" (UniqueName: \"kubernetes.io/projected/32a4c593-1f5c-4060-a418-f216a4bc105f-kube-api-access-4wpqk\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:42.980001 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:42.979960    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32a4c593-1f5c-4060-a418-f216a4bc105f-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:43.661000 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:43.660960    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e7414a-ndg6b" event={"ID":"32a4c593-1f5c-4060-a418-f216a4bc105f","Type":"ContainerDied","Data":"cc60e644e79ec0d0c040d8e88299ebab1b3da22ac9da3cef8dda70bb4aab152a"}
Oct 30 20:45:43.661185 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:43.661020    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc60e644e79ec0d0c040d8e88299ebab1b3da22ac9da3cef8dda70bb4aab152a"
Oct 30 20:45:43.661185 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:43.660984    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31e7414a-ndg6b"
Oct 30 20:45:45.693462 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:45.693426    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:46.729351 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:45:46.729316    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:45:46.866324 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:46.866290    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:47.262119 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:47.262070    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:51.176100 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:51.176056    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:51.720248 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:51.720216    5071 generic.go:356] "Generic (PLEG): container finished" podID="95b944d4-560b-48b9-929e-df69e4d9a213" containerID="296bea8c06c7eb492b5310ef5b1b92190581241c499fcf7b5e8fcf0fb6051386" exitCode=0
Oct 30 20:45:51.720401 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:51.720289    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-m6n4v-pdhcx" event={"ID":"95b944d4-560b-48b9-929e-df69e4d9a213","Type":"ContainerDied","Data":"296bea8c06c7eb492b5310ef5b1b92190581241c499fcf7b5e8fcf0fb6051386"}
Oct 30 20:45:52.525725 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:52.525692    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:52.920355 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:52.920331    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-m6n4v-pdhcx"
Oct 30 20:45:53.054227 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:53.054138    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95b944d4-560b-48b9-929e-df69e4d9a213-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/95b944d4-560b-48b9-929e-df69e4d9a213-archives-path\") pod \"95b944d4-560b-48b9-929e-df69e4d9a213\" (UID: \"95b944d4-560b-48b9-929e-df69e4d9a213\") "
Oct 30 20:45:53.054227 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:53.054174    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95b944d4-560b-48b9-929e-df69e4d9a213-kube-api-access-cztlh\" (UniqueName: \"kubernetes.io/projected/95b944d4-560b-48b9-929e-df69e4d9a213-kube-api-access-cztlh\") pod \"95b944d4-560b-48b9-929e-df69e4d9a213\" (UID: \"95b944d4-560b-48b9-929e-df69e4d9a213\") "
Oct 30 20:45:53.054461 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:53.054330    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95b944d4-560b-48b9-929e-df69e4d9a213-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95b944d4-560b-48b9-929e-df69e4d9a213-service-ca-bundle\") pod \"95b944d4-560b-48b9-929e-df69e4d9a213\" (UID: \"95b944d4-560b-48b9-929e-df69e4d9a213\") "
Oct 30 20:45:53.054563 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:53.054543    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95b944d4-560b-48b9-929e-df69e4d9a213-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95b944d4-560b-48b9-929e-df69e4d9a213-trusted-ca-bundle\") pod \"95b944d4-560b-48b9-929e-df69e4d9a213\" (UID: \"95b944d4-560b-48b9-929e-df69e4d9a213\") "
Oct 30 20:45:53.054652 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:53.054576    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95b944d4-560b-48b9-929e-df69e4d9a213-archives-path" pod "95b944d4-560b-48b9-929e-df69e4d9a213" (UID: "95b944d4-560b-48b9-929e-df69e4d9a213"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:45:53.054934 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:53.054906    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95b944d4-560b-48b9-929e-df69e4d9a213-service-ca-bundle" pod "95b944d4-560b-48b9-929e-df69e4d9a213" (UID: "95b944d4-560b-48b9-929e-df69e4d9a213"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:45:53.055032 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:53.055020    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95b944d4-560b-48b9-929e-df69e4d9a213-trusted-ca-bundle" pod "95b944d4-560b-48b9-929e-df69e4d9a213" (UID: "95b944d4-560b-48b9-929e-df69e4d9a213"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:45:53.055298 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:53.055279    5071 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95b944d4-560b-48b9-929e-df69e4d9a213-service-ca-bundle\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:53.055377 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:53.055304    5071 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95b944d4-560b-48b9-929e-df69e4d9a213-trusted-ca-bundle\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:53.055377 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:53.055319    5071 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/95b944d4-560b-48b9-929e-df69e4d9a213-archives-path\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:53.057114 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:53.057092    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95b944d4-560b-48b9-929e-df69e4d9a213-kube-api-access-cztlh" pod "95b944d4-560b-48b9-929e-df69e4d9a213" (UID: "95b944d4-560b-48b9-929e-df69e4d9a213"). InnerVolumeSpecName "kube-api-access-cztlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:45:53.156916 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:53.156875    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cztlh\" (UniqueName: \"kubernetes.io/projected/95b944d4-560b-48b9-929e-df69e4d9a213-kube-api-access-cztlh\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:53.741961 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:53.741924    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-m6n4v-pdhcx" event={"ID":"95b944d4-560b-48b9-929e-df69e4d9a213","Type":"ContainerDied","Data":"f4d7fbed170f9485237f69dcf86297e9a94502e664ce198f508fcab7d61c455a"}
Oct 30 20:45:53.742353 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:53.741982    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4d7fbed170f9485237f69dcf86297e9a94502e664ce198f508fcab7d61c455a"
Oct 30 20:45:53.742353 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:53.741939    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-m6n4v-pdhcx"
Oct 30 20:45:54.892302 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:54.892264    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:55.099964 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:55.099932    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:55.318801 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:55.318773    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae05d1da-5jv5l"]
Oct 30 20:45:55.319411 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:55.319396    5071 cpu_manager.go:414] "containerMap: removing container" podUID="4ee26fb5-c5d7-4b70-a994-67b1c487e5da" containerName="cleanup"
Oct 30 20:45:55.319456 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:55.319414    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ee26fb5-c5d7-4b70-a994-67b1c487e5da" containerName="cleanup"
Oct 30 20:45:55.319456 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:55.319442    5071 cpu_manager.go:414] "containerMap: removing container" podUID="3892545f-5bd2-490b-bd69-b1e8434da80c" containerName="cleanup"
Oct 30 20:45:55.319456 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:55.319447    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3892545f-5bd2-490b-bd69-b1e8434da80c" containerName="cleanup"
Oct 30 20:45:55.319456 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:55.319454    5071 cpu_manager.go:414] "containerMap: removing container" podUID="26471d92-815c-471c-b8be-52f4fe79dbc8" containerName="cleanup"
Oct 30 20:45:55.319583 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:55.319459    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26471d92-815c-471c-b8be-52f4fe79dbc8" containerName="cleanup"
Oct 30 20:45:55.319583 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:55.319483    5071 cpu_manager.go:414] "containerMap: removing container" podUID="32a4c593-1f5c-4060-a418-f216a4bc105f" containerName="revoke-keys"
Oct 30 20:45:55.319583 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:55.319488    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32a4c593-1f5c-4060-a418-f216a4bc105f" containerName="revoke-keys"
Oct 30 20:45:55.319583 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:55.319519    5071 cpu_manager.go:414] "containerMap: removing container" podUID="95b944d4-560b-48b9-929e-df69e4d9a213" containerName="insights-gathering"
Oct 30 20:45:55.319583 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:55.319523    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95b944d4-560b-48b9-929e-df69e4d9a213" containerName="insights-gathering"
Oct 30 20:45:55.319727 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:55.319642    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="95b944d4-560b-48b9-929e-df69e4d9a213" containerName="insights-gathering"
Oct 30 20:45:55.319727 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:55.319650    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="3892545f-5bd2-490b-bd69-b1e8434da80c" containerName="cleanup"
Oct 30 20:45:55.319727 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:55.319657    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ee26fb5-c5d7-4b70-a994-67b1c487e5da" containerName="cleanup"
Oct 30 20:45:55.319727 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:55.319664    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="26471d92-815c-471c-b8be-52f4fe79dbc8" containerName="cleanup"
Oct 30 20:45:55.319727 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:55.319671    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="0729ed1c-bbfd-48a8-97aa-f2fe75735a9e" containerName="cleanup"
Oct 30 20:45:55.319727 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:55.319679    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="32a4c593-1f5c-4060-a418-f216a4bc105f" containerName="revoke-keys"
Oct 30 20:45:55.323527 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:55.323507    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae05d1da-5jv5l"
Oct 30 20:45:55.381778 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:55.381749    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae05d1da-5jv5l"]
Oct 30 20:45:55.425665 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:55.425628    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:45:55.425851 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:55.425689    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 20:45:55.491479 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:55.491448    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10967423-677a-4cfe-ab12-8cda6ebac21a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae05d1da-5jv5l\" (UID: \"10967423-677a-4cfe-ab12-8cda6ebac21a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae05d1da-5jv5l"
Oct 30 20:45:55.491676 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:55.491514    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck7qt\" (UniqueName: \"kubernetes.io/projected/10967423-677a-4cfe-ab12-8cda6ebac21a-kube-api-access-ck7qt\") pod \"maas-api-revoke-sub-ae05d1da-5jv5l\" (UID: \"10967423-677a-4cfe-ab12-8cda6ebac21a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae05d1da-5jv5l"
Oct 30 20:45:55.491769 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:55.491749    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10967423-677a-4cfe-ab12-8cda6ebac21a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae05d1da-5jv5l\" (UID: \"10967423-677a-4cfe-ab12-8cda6ebac21a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae05d1da-5jv5l"
Oct 30 20:45:55.593342 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:55.593257    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10967423-677a-4cfe-ab12-8cda6ebac21a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae05d1da-5jv5l\" (UID: \"10967423-677a-4cfe-ab12-8cda6ebac21a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae05d1da-5jv5l"
Oct 30 20:45:55.593482 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:55.593380    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10967423-677a-4cfe-ab12-8cda6ebac21a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae05d1da-5jv5l\" (UID: \"10967423-677a-4cfe-ab12-8cda6ebac21a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae05d1da-5jv5l"
Oct 30 20:45:55.593482 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:55.593415    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck7qt\" (UniqueName: \"kubernetes.io/projected/10967423-677a-4cfe-ab12-8cda6ebac21a-kube-api-access-ck7qt\") pod \"maas-api-revoke-sub-ae05d1da-5jv5l\" (UID: \"10967423-677a-4cfe-ab12-8cda6ebac21a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae05d1da-5jv5l"
Oct 30 20:45:55.594180 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:55.594096    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10967423-677a-4cfe-ab12-8cda6ebac21a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae05d1da-5jv5l\" (UID: \"10967423-677a-4cfe-ab12-8cda6ebac21a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae05d1da-5jv5l"
Oct 30 20:45:55.595935 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:55.595910    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10967423-677a-4cfe-ab12-8cda6ebac21a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae05d1da-5jv5l\" (UID: \"10967423-677a-4cfe-ab12-8cda6ebac21a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae05d1da-5jv5l"
Oct 30 20:45:55.652034 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:55.651997    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck7qt\" (UniqueName: \"kubernetes.io/projected/10967423-677a-4cfe-ab12-8cda6ebac21a-kube-api-access-ck7qt\") pod \"maas-api-revoke-sub-ae05d1da-5jv5l\" (UID: \"10967423-677a-4cfe-ab12-8cda6ebac21a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae05d1da-5jv5l"
Oct 30 20:45:55.938032 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:55.937940    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae05d1da-5jv5l"
Oct 30 20:45:56.323160 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:45:56.323132    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10967423_677a_4cfe_ab12_8cda6ebac21a.slice/crio-2f42153f5a60481db262002f2eebe381fd76165ab6f36275d3febd38a3e1e067 WatchSource:0}: Error finding container 2f42153f5a60481db262002f2eebe381fd76165ab6f36275d3febd38a3e1e067: Status 404 returned error can't find the container with id 2f42153f5a60481db262002f2eebe381fd76165ab6f36275d3febd38a3e1e067
Oct 30 20:45:56.323893 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:56.323864    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae05d1da-5jv5l"]
Oct 30 20:45:56.769279 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:56.769239    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae05d1da-5jv5l" event={"ID":"10967423-677a-4cfe-ab12-8cda6ebac21a","Type":"ContainerStarted","Data":"98533b63626ff9bb5f3eddee40cec6509b742f0c9259197b902ee0014c8c1560"}
Oct 30 20:45:56.769445 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:56.769302    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae05d1da-5jv5l" event={"ID":"10967423-677a-4cfe-ab12-8cda6ebac21a","Type":"ContainerStarted","Data":"2f42153f5a60481db262002f2eebe381fd76165ab6f36275d3febd38a3e1e067"}
Oct 30 20:45:56.895216 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:56.895107    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae05d1da-5jv5l" podStartSLOduration=1.8950920500000001 podStartE2EDuration="1.89509205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:56.892678654 +0000 UTC m=+1168.809700194" watchObservedRunningTime="2026-10-30 20:45:56.89509205 +0000 UTC m=+1168.812113586"
Oct 30 20:45:57.313033 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:57.312880    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:45:57.778616 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:57.778585    5071 generic.go:356] "Generic (PLEG): container finished" podID="10967423-677a-4cfe-ab12-8cda6ebac21a" containerID="98533b63626ff9bb5f3eddee40cec6509b742f0c9259197b902ee0014c8c1560" exitCode=0
Oct 30 20:45:57.778848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:57.778680    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae05d1da-5jv5l" event={"ID":"10967423-677a-4cfe-ab12-8cda6ebac21a","Type":"ContainerDied","Data":"98533b63626ff9bb5f3eddee40cec6509b742f0c9259197b902ee0014c8c1560"}
Oct 30 20:45:59.201797 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:59.201767    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae05d1da-5jv5l"
Oct 30 20:45:59.418899 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:59.391384    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10967423-677a-4cfe-ab12-8cda6ebac21a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10967423-677a-4cfe-ab12-8cda6ebac21a-maas-api-openshift-ca\") pod \"10967423-677a-4cfe-ab12-8cda6ebac21a\" (UID: \"10967423-677a-4cfe-ab12-8cda6ebac21a\") "
Oct 30 20:45:59.418899 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:59.391611    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10967423-677a-4cfe-ab12-8cda6ebac21a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10967423-677a-4cfe-ab12-8cda6ebac21a-maas-api-xks-ca\") pod \"10967423-677a-4cfe-ab12-8cda6ebac21a\" (UID: \"10967423-677a-4cfe-ab12-8cda6ebac21a\") "
Oct 30 20:45:59.418899 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:59.391660    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10967423-677a-4cfe-ab12-8cda6ebac21a-kube-api-access-ck7qt\" (UniqueName: \"kubernetes.io/projected/10967423-677a-4cfe-ab12-8cda6ebac21a-kube-api-access-ck7qt\") pod \"10967423-677a-4cfe-ab12-8cda6ebac21a\" (UID: \"10967423-677a-4cfe-ab12-8cda6ebac21a\") "
Oct 30 20:45:59.418899 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:59.392596    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10967423-677a-4cfe-ab12-8cda6ebac21a-maas-api-openshift-ca" pod "10967423-677a-4cfe-ab12-8cda6ebac21a" (UID: "10967423-677a-4cfe-ab12-8cda6ebac21a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:45:59.483887 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:59.444149    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10967423-677a-4cfe-ab12-8cda6ebac21a-maas-api-xks-ca" pod "10967423-677a-4cfe-ab12-8cda6ebac21a" (UID: "10967423-677a-4cfe-ab12-8cda6ebac21a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:45:59.567849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:59.514647    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10967423-677a-4cfe-ab12-8cda6ebac21a-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:59.569354 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:59.568700    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10967423-677a-4cfe-ab12-8cda6ebac21a-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:59.814475 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:59.814432    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10967423-677a-4cfe-ab12-8cda6ebac21a-kube-api-access-ck7qt" pod "10967423-677a-4cfe-ab12-8cda6ebac21a" (UID: "10967423-677a-4cfe-ab12-8cda6ebac21a"). InnerVolumeSpecName "kube-api-access-ck7qt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:45:59.825337 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:59.825164    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck7qt\" (UniqueName: \"kubernetes.io/projected/10967423-677a-4cfe-ab12-8cda6ebac21a-kube-api-access-ck7qt\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:59.896713 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:59.896683    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae05d1da-5jv5l"
Oct 30 20:45:59.912080 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:59.912027    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae05d1da-5jv5l" event={"ID":"10967423-677a-4cfe-ab12-8cda6ebac21a","Type":"ContainerDied","Data":"2f42153f5a60481db262002f2eebe381fd76165ab6f36275d3febd38a3e1e067"}
Oct 30 20:45:59.912233 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:45:59.912144    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f42153f5a60481db262002f2eebe381fd76165ab6f36275d3febd38a3e1e067"
Oct 30 20:46:02.484617 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:02.484580    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:06.060960 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:06.060924    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-683f3132-5bq2k"]
Oct 30 20:46:06.108150 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:06.108118    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-683f3132-5bq2k"]
Oct 30 20:46:06.202120 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:06.202088    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55" path="/var/lib/kubelet/pods/9e7aa51a-7ae4-4ba1-a4c7-1079c70f3b55/volumes"
Oct 30 20:46:07.391667 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:07.391621    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:12.387335 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:12.387292    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:14.444379 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.444348    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"]
Oct 30 20:46:14.445019 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.445004    5071 cpu_manager.go:414] "containerMap: removing container" podUID="10967423-677a-4cfe-ab12-8cda6ebac21a" containerName="revoke-keys"
Oct 30 20:46:14.445086 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.445021    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10967423-677a-4cfe-ab12-8cda6ebac21a" containerName="revoke-keys"
Oct 30 20:46:14.445121 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.445108    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="10967423-677a-4cfe-ab12-8cda6ebac21a" containerName="revoke-keys"
Oct 30 20:46:14.448643 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.448625    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"
Oct 30 20:46:14.487880 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.487853    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7299979e-94b4-4d26-a860-c88b6136faf2-istio-envoy\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts\" (UID: \"7299979e-94b4-4d26-a860-c88b6136faf2\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"
Oct 30 20:46:14.488066 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.488004    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7299979e-94b4-4d26-a860-c88b6136faf2-istio-token\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts\" (UID: \"7299979e-94b4-4d26-a860-c88b6136faf2\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"
Oct 30 20:46:14.488159 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.488136    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7299979e-94b4-4d26-a860-c88b6136faf2-workload-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts\" (UID: \"7299979e-94b4-4d26-a860-c88b6136faf2\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"
Oct 30 20:46:14.488220 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.488171    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7299979e-94b4-4d26-a860-c88b6136faf2-workload-certs\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts\" (UID: \"7299979e-94b4-4d26-a860-c88b6136faf2\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"
Oct 30 20:46:14.488284 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.488236    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7299979e-94b4-4d26-a860-c88b6136faf2-credential-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts\" (UID: \"7299979e-94b4-4d26-a860-c88b6136faf2\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"
Oct 30 20:46:14.488284 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.488269    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7299979e-94b4-4d26-a860-c88b6136faf2-istio-data\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts\" (UID: \"7299979e-94b4-4d26-a860-c88b6136faf2\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"
Oct 30 20:46:14.488379 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.488296    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zj6b\" (UniqueName: \"kubernetes.io/projected/7299979e-94b4-4d26-a860-c88b6136faf2-kube-api-access-2zj6b\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts\" (UID: \"7299979e-94b4-4d26-a860-c88b6136faf2\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"
Oct 30 20:46:14.488452 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.488439    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7299979e-94b4-4d26-a860-c88b6136faf2-istio-podinfo\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts\" (UID: \"7299979e-94b4-4d26-a860-c88b6136faf2\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"
Oct 30 20:46:14.488502 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.488462    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7299979e-94b4-4d26-a860-c88b6136faf2-istiod-ca-cert\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts\" (UID: \"7299979e-94b4-4d26-a860-c88b6136faf2\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"
Oct 30 20:46:14.558465 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.558427    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"]
Oct 30 20:46:14.589703 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.589673    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7299979e-94b4-4d26-a860-c88b6136faf2-istio-podinfo\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts\" (UID: \"7299979e-94b4-4d26-a860-c88b6136faf2\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"
Oct 30 20:46:14.589703 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.589706    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7299979e-94b4-4d26-a860-c88b6136faf2-istiod-ca-cert\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts\" (UID: \"7299979e-94b4-4d26-a860-c88b6136faf2\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"
Oct 30 20:46:14.589943 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.589848    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7299979e-94b4-4d26-a860-c88b6136faf2-istio-envoy\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts\" (UID: \"7299979e-94b4-4d26-a860-c88b6136faf2\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"
Oct 30 20:46:14.589943 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.589935    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7299979e-94b4-4d26-a860-c88b6136faf2-istio-token\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts\" (UID: \"7299979e-94b4-4d26-a860-c88b6136faf2\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"
Oct 30 20:46:14.590061 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.590006    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7299979e-94b4-4d26-a860-c88b6136faf2-workload-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts\" (UID: \"7299979e-94b4-4d26-a860-c88b6136faf2\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"
Oct 30 20:46:14.590061 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.590022    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7299979e-94b4-4d26-a860-c88b6136faf2-workload-certs\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts\" (UID: \"7299979e-94b4-4d26-a860-c88b6136faf2\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"
Oct 30 20:46:14.590061 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.590040    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7299979e-94b4-4d26-a860-c88b6136faf2-credential-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts\" (UID: \"7299979e-94b4-4d26-a860-c88b6136faf2\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"
Oct 30 20:46:14.590061 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.590057    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7299979e-94b4-4d26-a860-c88b6136faf2-istio-data\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts\" (UID: \"7299979e-94b4-4d26-a860-c88b6136faf2\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"
Oct 30 20:46:14.590267 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.590071    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zj6b\" (UniqueName: \"kubernetes.io/projected/7299979e-94b4-4d26-a860-c88b6136faf2-kube-api-access-2zj6b\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts\" (UID: \"7299979e-94b4-4d26-a860-c88b6136faf2\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"
Oct 30 20:46:14.590507 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.590482    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7299979e-94b4-4d26-a860-c88b6136faf2-workload-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts\" (UID: \"7299979e-94b4-4d26-a860-c88b6136faf2\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"
Oct 30 20:46:14.590638 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.590510    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7299979e-94b4-4d26-a860-c88b6136faf2-istiod-ca-cert\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts\" (UID: \"7299979e-94b4-4d26-a860-c88b6136faf2\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"
Oct 30 20:46:14.590638 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.590548    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7299979e-94b4-4d26-a860-c88b6136faf2-workload-certs\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts\" (UID: \"7299979e-94b4-4d26-a860-c88b6136faf2\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"
Oct 30 20:46:14.590638 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.590581    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7299979e-94b4-4d26-a860-c88b6136faf2-credential-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts\" (UID: \"7299979e-94b4-4d26-a860-c88b6136faf2\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"
Oct 30 20:46:14.590778 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.590660    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7299979e-94b4-4d26-a860-c88b6136faf2-istio-data\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts\" (UID: \"7299979e-94b4-4d26-a860-c88b6136faf2\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"
Oct 30 20:46:14.593015 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.592986    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7299979e-94b4-4d26-a860-c88b6136faf2-istio-envoy\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts\" (UID: \"7299979e-94b4-4d26-a860-c88b6136faf2\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"
Oct 30 20:46:14.593141 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.593125    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7299979e-94b4-4d26-a860-c88b6136faf2-istio-podinfo\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts\" (UID: \"7299979e-94b4-4d26-a860-c88b6136faf2\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"
Oct 30 20:46:14.653339 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.653312    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7299979e-94b4-4d26-a860-c88b6136faf2-istio-token\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts\" (UID: \"7299979e-94b4-4d26-a860-c88b6136faf2\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"
Oct 30 20:46:14.653573 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.653552    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zj6b\" (UniqueName: \"kubernetes.io/projected/7299979e-94b4-4d26-a860-c88b6136faf2-kube-api-access-2zj6b\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts\" (UID: \"7299979e-94b4-4d26-a860-c88b6136faf2\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"
Oct 30 20:46:14.699734 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.699668    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"]
Oct 30 20:46:14.704981 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.704962    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"
Oct 30 20:46:14.773230 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.773195    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"
Oct 30 20:46:14.779088 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.779061    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"]
Oct 30 20:46:14.796910 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.796872    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09665d83-c120-4c47-bc5f-45fa16149bb4-istiod-ca-cert\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj\" (UID: \"09665d83-c120-4c47-bc5f-45fa16149bb4\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"
Oct 30 20:46:14.797081 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.797014    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09665d83-c120-4c47-bc5f-45fa16149bb4-istio-data\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj\" (UID: \"09665d83-c120-4c47-bc5f-45fa16149bb4\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"
Oct 30 20:46:14.797176 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.797156    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09665d83-c120-4c47-bc5f-45fa16149bb4-credential-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj\" (UID: \"09665d83-c120-4c47-bc5f-45fa16149bb4\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"
Oct 30 20:46:14.797320 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.797298    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg9ld\" (UniqueName: \"kubernetes.io/projected/09665d83-c120-4c47-bc5f-45fa16149bb4-kube-api-access-qg9ld\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj\" (UID: \"09665d83-c120-4c47-bc5f-45fa16149bb4\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"
Oct 30 20:46:14.797493 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.797469    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09665d83-c120-4c47-bc5f-45fa16149bb4-workload-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj\" (UID: \"09665d83-c120-4c47-bc5f-45fa16149bb4\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"
Oct 30 20:46:14.797689 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.797668    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09665d83-c120-4c47-bc5f-45fa16149bb4-workload-certs\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj\" (UID: \"09665d83-c120-4c47-bc5f-45fa16149bb4\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"
Oct 30 20:46:14.797889 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.797871    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09665d83-c120-4c47-bc5f-45fa16149bb4-istio-podinfo\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj\" (UID: \"09665d83-c120-4c47-bc5f-45fa16149bb4\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"
Oct 30 20:46:14.798101 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.798068    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09665d83-c120-4c47-bc5f-45fa16149bb4-istio-envoy\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj\" (UID: \"09665d83-c120-4c47-bc5f-45fa16149bb4\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"
Oct 30 20:46:14.798355 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.798339    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09665d83-c120-4c47-bc5f-45fa16149bb4-istio-token\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj\" (UID: \"09665d83-c120-4c47-bc5f-45fa16149bb4\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"
Oct 30 20:46:14.900041 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.900000    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg9ld\" (UniqueName: \"kubernetes.io/projected/09665d83-c120-4c47-bc5f-45fa16149bb4-kube-api-access-qg9ld\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj\" (UID: \"09665d83-c120-4c47-bc5f-45fa16149bb4\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"
Oct 30 20:46:14.900041 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.900039    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09665d83-c120-4c47-bc5f-45fa16149bb4-workload-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj\" (UID: \"09665d83-c120-4c47-bc5f-45fa16149bb4\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"
Oct 30 20:46:14.900309 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.900062    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09665d83-c120-4c47-bc5f-45fa16149bb4-workload-certs\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj\" (UID: \"09665d83-c120-4c47-bc5f-45fa16149bb4\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"
Oct 30 20:46:14.900369 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.900333    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09665d83-c120-4c47-bc5f-45fa16149bb4-istio-podinfo\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj\" (UID: \"09665d83-c120-4c47-bc5f-45fa16149bb4\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"
Oct 30 20:46:14.900429 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.900388    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09665d83-c120-4c47-bc5f-45fa16149bb4-istio-envoy\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj\" (UID: \"09665d83-c120-4c47-bc5f-45fa16149bb4\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"
Oct 30 20:46:14.900429 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.900411    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09665d83-c120-4c47-bc5f-45fa16149bb4-workload-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj\" (UID: \"09665d83-c120-4c47-bc5f-45fa16149bb4\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"
Oct 30 20:46:14.900533 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.900521    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09665d83-c120-4c47-bc5f-45fa16149bb4-istio-token\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj\" (UID: \"09665d83-c120-4c47-bc5f-45fa16149bb4\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"
Oct 30 20:46:14.900587 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.900564    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09665d83-c120-4c47-bc5f-45fa16149bb4-istiod-ca-cert\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj\" (UID: \"09665d83-c120-4c47-bc5f-45fa16149bb4\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"
Oct 30 20:46:14.900643 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.900602    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09665d83-c120-4c47-bc5f-45fa16149bb4-istio-data\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj\" (UID: \"09665d83-c120-4c47-bc5f-45fa16149bb4\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"
Oct 30 20:46:14.900643 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.900606    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09665d83-c120-4c47-bc5f-45fa16149bb4-workload-certs\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj\" (UID: \"09665d83-c120-4c47-bc5f-45fa16149bb4\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"
Oct 30 20:46:14.900746 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.900688    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09665d83-c120-4c47-bc5f-45fa16149bb4-credential-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj\" (UID: \"09665d83-c120-4c47-bc5f-45fa16149bb4\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"
Oct 30 20:46:14.901137 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.901090    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09665d83-c120-4c47-bc5f-45fa16149bb4-istio-data\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj\" (UID: \"09665d83-c120-4c47-bc5f-45fa16149bb4\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"
Oct 30 20:46:14.901137 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.901119    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09665d83-c120-4c47-bc5f-45fa16149bb4-credential-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj\" (UID: \"09665d83-c120-4c47-bc5f-45fa16149bb4\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"
Oct 30 20:46:14.901536 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.901511    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09665d83-c120-4c47-bc5f-45fa16149bb4-istiod-ca-cert\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj\" (UID: \"09665d83-c120-4c47-bc5f-45fa16149bb4\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"
Oct 30 20:46:14.903113 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.903090    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09665d83-c120-4c47-bc5f-45fa16149bb4-istio-podinfo\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj\" (UID: \"09665d83-c120-4c47-bc5f-45fa16149bb4\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"
Oct 30 20:46:14.903250 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.903231    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09665d83-c120-4c47-bc5f-45fa16149bb4-istio-envoy\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj\" (UID: \"09665d83-c120-4c47-bc5f-45fa16149bb4\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"
Oct 30 20:46:14.963969 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.963897    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09665d83-c120-4c47-bc5f-45fa16149bb4-istio-token\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj\" (UID: \"09665d83-c120-4c47-bc5f-45fa16149bb4\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"
Oct 30 20:46:14.964592 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:14.964556    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg9ld\" (UniqueName: \"kubernetes.io/projected/09665d83-c120-4c47-bc5f-45fa16149bb4-kube-api-access-qg9ld\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj\" (UID: \"09665d83-c120-4c47-bc5f-45fa16149bb4\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"
Oct 30 20:46:15.022743 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:15.022706    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"
Oct 30 20:46:15.242229 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:46:15.242189    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7299979e_94b4_4d26_a860_c88b6136faf2.slice/crio-cebc659d2daf172e099bc338d4f74a23f1d68bdafcf7da4364202b5da3afbd9b WatchSource:0}: Error finding container cebc659d2daf172e099bc338d4f74a23f1d68bdafcf7da4364202b5da3afbd9b: Status 404 returned error can't find the container with id cebc659d2daf172e099bc338d4f74a23f1d68bdafcf7da4364202b5da3afbd9b
Oct 30 20:46:15.243258 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:15.243234    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"]
Oct 30 20:46:15.244542 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:15.244511    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:46:15.244605 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:15.244589    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:46:15.244644 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:15.244620    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:46:15.411514 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:46:15.409158    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09665d83_c120_4c47_bc5f_45fa16149bb4.slice/crio-da8f340962856d5869435ab11ff364cbb3530d1b7cb6dd6d70eb748ef8c2e8f5 WatchSource:0}: Error finding container da8f340962856d5869435ab11ff364cbb3530d1b7cb6dd6d70eb748ef8c2e8f5: Status 404 returned error can't find the container with id da8f340962856d5869435ab11ff364cbb3530d1b7cb6dd6d70eb748ef8c2e8f5
Oct 30 20:46:15.411514 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:15.410552    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"]
Oct 30 20:46:15.411931 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:15.411727    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:46:15.411931 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:15.411785    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:46:15.411931 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:15.411851    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:46:16.083710 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.083655    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj" event={"ID":"09665d83-c120-4c47-bc5f-45fa16149bb4","Type":"ContainerStarted","Data":"bf58317edd0ff572f75077ded61beb7eec136964bb1ece6e5c84a6a87cd40869"}
Oct 30 20:46:16.084338 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.083726    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj" event={"ID":"09665d83-c120-4c47-bc5f-45fa16149bb4","Type":"ContainerStarted","Data":"da8f340962856d5869435ab11ff364cbb3530d1b7cb6dd6d70eb748ef8c2e8f5"}
Oct 30 20:46:16.087205 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.087177    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts" event={"ID":"7299979e-94b4-4d26-a860-c88b6136faf2","Type":"ContainerStarted","Data":"78f1719a501680ab87a17a2667dbae3db84a6f30a1c9aa976c4a04e878861d1a"}
Oct 30 20:46:16.087349 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.087220    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts" event={"ID":"7299979e-94b4-4d26-a860-c88b6136faf2","Type":"ContainerStarted","Data":"cebc659d2daf172e099bc338d4f74a23f1d68bdafcf7da4364202b5da3afbd9b"}
Oct 30 20:46:16.210538 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.210487    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj" podStartSLOduration=2.210473514 podStartE2EDuration="2.210473514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:16.20675813 +0000 UTC m=+1188.123779661" watchObservedRunningTime="2026-10-30 20:46:16.210473514 +0000 UTC m=+1188.127495097"
Oct 30 20:46:16.595971 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.595914    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts" podStartSLOduration=2.595900363 podStartE2EDuration="2.595900363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:16.365315857 +0000 UTC m=+1188.282337394" watchObservedRunningTime="2026-10-30 20:46:16.595900363 +0000 UTC m=+1188.512921899"
Oct 30 20:46:16.597216 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.597196    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"]
Oct 30 20:46:16.601531 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.601514    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"
Oct 30 20:46:16.624105 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.624078    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-credential-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp\" (UID: \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"
Oct 30 20:46:16.624212 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.624114    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-istiod-ca-cert\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp\" (UID: \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"
Oct 30 20:46:16.624212 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.624198    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2g7d\" (UniqueName: \"kubernetes.io/projected/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-kube-api-access-k2g7d\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp\" (UID: \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"
Oct 30 20:46:16.624306 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.624234    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-istio-podinfo\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp\" (UID: \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"
Oct 30 20:46:16.624358 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.624339    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-istio-data\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp\" (UID: \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"
Oct 30 20:46:16.624399 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.624378    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-istio-token\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp\" (UID: \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"
Oct 30 20:46:16.624483 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.624468    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-workload-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp\" (UID: \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"
Oct 30 20:46:16.624530 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.624513    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-istio-envoy\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp\" (UID: \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"
Oct 30 20:46:16.624592 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.624581    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-workload-certs\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp\" (UID: \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"
Oct 30 20:46:16.728311 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.727366    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-istio-data\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp\" (UID: \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"
Oct 30 20:46:16.728311 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.727452    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-istio-token\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp\" (UID: \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"
Oct 30 20:46:16.728311 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.727595    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-workload-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp\" (UID: \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"
Oct 30 20:46:16.728311 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.727630    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-istio-envoy\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp\" (UID: \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"
Oct 30 20:46:16.728311 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.727789    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-workload-certs\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp\" (UID: \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"
Oct 30 20:46:16.747858 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.747633    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-workload-certs\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp\" (UID: \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"
Oct 30 20:46:16.747858 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.747811    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-credential-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp\" (UID: \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"
Oct 30 20:46:16.748109 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.747882    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-istiod-ca-cert\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp\" (UID: \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"
Oct 30 20:46:16.748109 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.747940    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2g7d\" (UniqueName: \"kubernetes.io/projected/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-kube-api-access-k2g7d\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp\" (UID: \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"
Oct 30 20:46:16.748109 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.747997    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-istio-podinfo\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp\" (UID: \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"
Oct 30 20:46:16.756854 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.749135    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"]
Oct 30 20:46:16.756854 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.749440    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-workload-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp\" (UID: \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"
Oct 30 20:46:16.756854 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.750088    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-istiod-ca-cert\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp\" (UID: \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"
Oct 30 20:46:16.756854 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.750342    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-credential-socket\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp\" (UID: \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"
Oct 30 20:46:16.756854 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.750576    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-istio-data\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp\" (UID: \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"
Oct 30 20:46:16.791848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.787934    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"
Oct 30 20:46:16.808355 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.798076    5071 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 20:46:16.808355 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.798186    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts" podUID="7299979e-94b4-4d26-a860-c88b6136faf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 20:46:16.808355 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.806077    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-istio-podinfo\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp\" (UID: \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"
Oct 30 20:46:16.820000 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.809095    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-istio-envoy\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp\" (UID: \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"
Oct 30 20:46:16.820000 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.817911    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2g7d\" (UniqueName: \"kubernetes.io/projected/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-kube-api-access-k2g7d\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp\" (UID: \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"
Oct 30 20:46:16.900757 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.896446    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-istio-token\") pod \"e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp\" (UID: \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\") " pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"
Oct 30 20:46:16.906001 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:46:16.905936    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:46:16.916778 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:16.916756    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"
Oct 30 20:46:17.024490 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:17.023808    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"
Oct 30 20:46:17.025575 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:17.025545    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 20:46:17.025725 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:17.025591    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj" podUID="09665d83-c120-4c47-bc5f-45fa16149bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 20:46:17.500270 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:17.498810    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:46:17.500270 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:17.498933    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:46:17.500270 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:17.498979    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:46:17.504953 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:17.504930    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"]
Oct 30 20:46:17.568847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:17.560317    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:17.806774 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:17.806738    5071 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 20:46:17.807061 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:17.807030    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts" podUID="7299979e-94b4-4d26-a860-c88b6136faf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 20:46:18.025023 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:18.024983    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 20:46:18.025290 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:18.025258    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj" podUID="09665d83-c120-4c47-bc5f-45fa16149bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 20:46:18.108995 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:18.108908    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp" event={"ID":"cefe43d8-48c1-4ba9-923c-3c2da5db8f01","Type":"ContainerStarted","Data":"73abc8b1e0fe5fc25518ba3aa5acec46892d19c98ee6ee9a929d078bd526ba7c"}
Oct 30 20:46:18.108995 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:18.108955    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp" event={"ID":"cefe43d8-48c1-4ba9-923c-3c2da5db8f01","Type":"ContainerStarted","Data":"c92d22b26707cfd5bebebfba86cf1709beae8036b9bb0d124bf41aa1a9d0e39f"}
Oct 30 20:46:18.258476 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:18.258408    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp" podStartSLOduration=2.258387586 podStartE2EDuration="2.258387586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:18.247178461 +0000 UTC m=+1190.164199998" watchObservedRunningTime="2026-10-30 20:46:18.258387586 +0000 UTC m=+1190.175409122"
Oct 30 20:46:18.790847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:18.783229    5071 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 20:46:18.790847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:18.783311    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts" podUID="7299979e-94b4-4d26-a860-c88b6136faf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 20:46:18.917473 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:18.917432    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"
Oct 30 20:46:18.918853 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:18.918810    5071 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 20:46:18.918909 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:18.918883    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp" podUID="cefe43d8-48c1-4ba9-923c-3c2da5db8f01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 20:46:19.023438 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:19.023396    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 20:46:19.023645 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:19.023445    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj" podUID="09665d83-c120-4c47-bc5f-45fa16149bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 20:46:19.777850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:19.777213    5071 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 20:46:19.777850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:19.777290    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts" podUID="7299979e-94b4-4d26-a860-c88b6136faf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 20:46:19.917776 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:19.917741    5071 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 20:46:19.918180 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:19.917798    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp" podUID="cefe43d8-48c1-4ba9-923c-3c2da5db8f01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 20:46:20.023361 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:20.023321    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 20:46:20.023534 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:20.023373    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj" podUID="09665d83-c120-4c47-bc5f-45fa16149bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 20:46:20.085849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:20.085192    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36a4e29c-hhw62"]
Oct 30 20:46:20.109934 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:20.108232    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a4e29c-hhw62"
Oct 30 20:46:20.197848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:20.180768    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 20:46:20.197848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:20.180865    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:46:20.226447 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:20.226411    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36a4e29c-hhw62"]
Oct 30 20:46:20.305851 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:20.264062    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7474j\" (UniqueName: \"kubernetes.io/projected/51a78fcc-db75-4d29-8228-edeb380f36ec-kube-api-access-7474j\") pod \"maas-api-revoke-sub-36a4e29c-hhw62\" (UID: \"51a78fcc-db75-4d29-8228-edeb380f36ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a4e29c-hhw62"
Oct 30 20:46:20.317606 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:20.317561    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51a78fcc-db75-4d29-8228-edeb380f36ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36a4e29c-hhw62\" (UID: \"51a78fcc-db75-4d29-8228-edeb380f36ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a4e29c-hhw62"
Oct 30 20:46:20.317983 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:20.317962    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51a78fcc-db75-4d29-8228-edeb380f36ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36a4e29c-hhw62\" (UID: \"51a78fcc-db75-4d29-8228-edeb380f36ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a4e29c-hhw62"
Oct 30 20:46:20.419732 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:20.418937    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51a78fcc-db75-4d29-8228-edeb380f36ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36a4e29c-hhw62\" (UID: \"51a78fcc-db75-4d29-8228-edeb380f36ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a4e29c-hhw62"
Oct 30 20:46:20.419732 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:20.419054    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7474j\" (UniqueName: \"kubernetes.io/projected/51a78fcc-db75-4d29-8228-edeb380f36ec-kube-api-access-7474j\") pod \"maas-api-revoke-sub-36a4e29c-hhw62\" (UID: \"51a78fcc-db75-4d29-8228-edeb380f36ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a4e29c-hhw62"
Oct 30 20:46:20.419732 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:20.419124    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51a78fcc-db75-4d29-8228-edeb380f36ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36a4e29c-hhw62\" (UID: \"51a78fcc-db75-4d29-8228-edeb380f36ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a4e29c-hhw62"
Oct 30 20:46:20.420244 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:20.420220    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51a78fcc-db75-4d29-8228-edeb380f36ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36a4e29c-hhw62\" (UID: \"51a78fcc-db75-4d29-8228-edeb380f36ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a4e29c-hhw62"
Oct 30 20:46:20.421974 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:20.421949    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51a78fcc-db75-4d29-8228-edeb380f36ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36a4e29c-hhw62\" (UID: \"51a78fcc-db75-4d29-8228-edeb380f36ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a4e29c-hhw62"
Oct 30 20:46:20.512985 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:20.512951    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7474j\" (UniqueName: \"kubernetes.io/projected/51a78fcc-db75-4d29-8228-edeb380f36ec-kube-api-access-7474j\") pod \"maas-api-revoke-sub-36a4e29c-hhw62\" (UID: \"51a78fcc-db75-4d29-8228-edeb380f36ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a4e29c-hhw62"
Oct 30 20:46:20.726337 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:20.726264    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a4e29c-hhw62"
Oct 30 20:46:20.796081 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:20.789102    5071 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 20:46:20.796081 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:20.789158    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts" podUID="7299979e-94b4-4d26-a860-c88b6136faf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 20:46:20.923411 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:20.923360    5071 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 20:46:20.923982 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:20.923426    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp" podUID="cefe43d8-48c1-4ba9-923c-3c2da5db8f01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 20:46:21.026720 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:21.025162    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 20:46:21.026720 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:21.025218    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj" podUID="09665d83-c120-4c47-bc5f-45fa16149bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 20:46:21.223868 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:21.220433    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adc2708d-dbrhj"]
Oct 30 20:46:21.426862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:21.421296    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adc2708d-dbrhj"]
Oct 30 20:46:21.777951 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:21.777805    5071 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 20:46:21.777951 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:21.777915    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts" podUID="7299979e-94b4-4d26-a860-c88b6136faf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 20:46:21.788743 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:21.788708    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36a4e29c-hhw62"]
Oct 30 20:46:21.918271 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:21.918228    5071 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 20:46:21.918446 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:21.918288    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp" podUID="cefe43d8-48c1-4ba9-923c-3c2da5db8f01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 20:46:22.025265 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:22.024186    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 20:46:22.025265 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:22.024250    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj" podUID="09665d83-c120-4c47-bc5f-45fa16149bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 20:46:22.212864 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:22.210043    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="544da439-8c62-47c7-bdb5-256b8b55a890" path="/var/lib/kubelet/pods/544da439-8c62-47c7-bdb5-256b8b55a890/volumes"
Oct 30 20:46:22.452907 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:22.452039    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a4e29c-hhw62" event={"ID":"51a78fcc-db75-4d29-8228-edeb380f36ec","Type":"ContainerStarted","Data":"30131cdd25c32bb4cfc36f576f5c4d61b9ea299fa0543c7c6267c8c7cb03d543"}
Oct 30 20:46:22.452907 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:22.452110    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a4e29c-hhw62" event={"ID":"51a78fcc-db75-4d29-8228-edeb380f36ec","Type":"ContainerStarted","Data":"40f6d73d23f6039395518723dab77cdcad9e950c14f2a31a8e0b6c98281dd96a"}
Oct 30 20:46:22.577179 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:22.577113    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a4e29c-hhw62" podStartSLOduration=2.577092654 podStartE2EDuration="2.577092654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:22.568305772 +0000 UTC m=+1194.485327308" watchObservedRunningTime="2026-10-30 20:46:22.577092654 +0000 UTC m=+1194.494114190"
Oct 30 20:46:22.905058 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:22.905021    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:22.960652 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:22.957109    5071 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 20:46:22.960652 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:22.957175    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp" podUID="cefe43d8-48c1-4ba9-923c-3c2da5db8f01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 20:46:23.392496 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:23.391535    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"
Oct 30 20:46:23.393368 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:23.392646    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"
Oct 30 20:46:23.398868 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:23.398076    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"
Oct 30 20:46:23.495411 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:23.495366    5071 generic.go:356] "Generic (PLEG): container finished" podID="51a78fcc-db75-4d29-8228-edeb380f36ec" containerID="30131cdd25c32bb4cfc36f576f5c4d61b9ea299fa0543c7c6267c8c7cb03d543" exitCode=0
Oct 30 20:46:23.496598 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:23.496527    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a4e29c-hhw62" event={"ID":"51a78fcc-db75-4d29-8228-edeb380f36ec","Type":"ContainerDied","Data":"30131cdd25c32bb4cfc36f576f5c4d61b9ea299fa0543c7c6267c8c7cb03d543"}
Oct 30 20:46:23.512313 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:23.512073    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"
Oct 30 20:46:23.516081 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:23.512883    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"
Oct 30 20:46:23.516081 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:23.514429    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"
Oct 30 20:46:23.917891 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:23.917857    5071 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 20:46:23.918150 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:23.917920    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp" podUID="cefe43d8-48c1-4ba9-923c-3c2da5db8f01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 20:46:24.707593 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:24.707564    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a4e29c-hhw62"
Oct 30 20:46:24.777583 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:24.777496    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51a78fcc-db75-4d29-8228-edeb380f36ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51a78fcc-db75-4d29-8228-edeb380f36ec-maas-api-openshift-ca\") pod \"51a78fcc-db75-4d29-8228-edeb380f36ec\" (UID: \"51a78fcc-db75-4d29-8228-edeb380f36ec\") "
Oct 30 20:46:24.777583 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:24.777560    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51a78fcc-db75-4d29-8228-edeb380f36ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51a78fcc-db75-4d29-8228-edeb380f36ec-maas-api-xks-ca\") pod \"51a78fcc-db75-4d29-8228-edeb380f36ec\" (UID: \"51a78fcc-db75-4d29-8228-edeb380f36ec\") "
Oct 30 20:46:24.777779 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:24.777590    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51a78fcc-db75-4d29-8228-edeb380f36ec-kube-api-access-7474j\" (UniqueName: \"kubernetes.io/projected/51a78fcc-db75-4d29-8228-edeb380f36ec-kube-api-access-7474j\") pod \"51a78fcc-db75-4d29-8228-edeb380f36ec\" (UID: \"51a78fcc-db75-4d29-8228-edeb380f36ec\") "
Oct 30 20:46:24.777995 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:24.777967    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51a78fcc-db75-4d29-8228-edeb380f36ec-maas-api-openshift-ca" pod "51a78fcc-db75-4d29-8228-edeb380f36ec" (UID: "51a78fcc-db75-4d29-8228-edeb380f36ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:46:24.778154 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:24.778130    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51a78fcc-db75-4d29-8228-edeb380f36ec-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:24.780560 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:24.780532    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51a78fcc-db75-4d29-8228-edeb380f36ec-maas-api-xks-ca" pod "51a78fcc-db75-4d29-8228-edeb380f36ec" (UID: "51a78fcc-db75-4d29-8228-edeb380f36ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:46:24.780643 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:24.780568    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a78fcc-db75-4d29-8228-edeb380f36ec-kube-api-access-7474j" pod "51a78fcc-db75-4d29-8228-edeb380f36ec" (UID: "51a78fcc-db75-4d29-8228-edeb380f36ec"). InnerVolumeSpecName "kube-api-access-7474j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:46:24.879467 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:24.879424    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51a78fcc-db75-4d29-8228-edeb380f36ec-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:24.879467 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:24.879467    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7474j\" (UniqueName: \"kubernetes.io/projected/51a78fcc-db75-4d29-8228-edeb380f36ec-kube-api-access-7474j\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:25.403715 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:25.403351    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"
Oct 30 20:46:25.404295 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:25.404271    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"
Oct 30 20:46:25.405126 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:25.405106    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"
Oct 30 20:46:25.513070 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:25.513040    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a4e29c-hhw62"
Oct 30 20:46:25.513233 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:25.513125    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a4e29c-hhw62" event={"ID":"51a78fcc-db75-4d29-8228-edeb380f36ec","Type":"ContainerDied","Data":"40f6d73d23f6039395518723dab77cdcad9e950c14f2a31a8e0b6c98281dd96a"}
Oct 30 20:46:25.513233 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:25.513170    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40f6d73d23f6039395518723dab77cdcad9e950c14f2a31a8e0b6c98281dd96a"
Oct 30 20:46:26.066386 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:26.066351    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7b2ab73-wfflr"]
Oct 30 20:46:26.073216 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:26.073186    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7b2ab73-wfflr"]
Oct 30 20:46:26.203333 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:26.203300    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e12a218-b48c-46c0-a782-7d7036f83585" path="/var/lib/kubelet/pods/8e12a218-b48c-46c0-a782-7d7036f83585/volumes"
Oct 30 20:46:27.004771 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:27.004720    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:27.973745 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:27.973706    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:28.383848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:28.383803    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8qvb_91fe9e06-0c30-4000-bddb-2ba89cd048cc/console-operator/2.log"
Oct 30 20:46:28.386194 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:28.386167    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8qvb_91fe9e06-0c30-4000-bddb-2ba89cd048cc/console-operator/2.log"
Oct 30 20:46:28.392128 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:28.392109    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q4f4h_80cae79e-b60f-4efe-bb58-e77753a457e5/ovn-acl-logging/0.log"
Oct 30 20:46:28.393968 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:28.393936    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q4f4h_80cae79e-b60f-4efe-bb58-e77753a457e5/ovn-acl-logging/0.log"
Oct 30 20:46:29.101861 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:29.101801    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:29.961445 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:29.961409    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:31.382840 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:31.382797    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:46:33.384849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:33.383878    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:36.081650 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:36.081614    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb0460da-vdhn6"]
Oct 30 20:46:36.082309 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:36.082293    5071 cpu_manager.go:414] "containerMap: removing container" podUID="51a78fcc-db75-4d29-8228-edeb380f36ec" containerName="revoke-keys"
Oct 30 20:46:36.082354 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:36.082311    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51a78fcc-db75-4d29-8228-edeb380f36ec" containerName="revoke-keys"
Oct 30 20:46:36.082481 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:36.082472    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="51a78fcc-db75-4d29-8228-edeb380f36ec" containerName="revoke-keys"
Oct 30 20:46:36.087619 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:36.087601    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb0460da-vdhn6"
Oct 30 20:46:36.145356 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:36.145319    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb0460da-vdhn6"]
Oct 30 20:46:36.188192 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:36.188154    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:46:36.188192 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:36.188178    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 20:46:36.203766 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:36.203739    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b54be71f-g6w4m"]
Oct 30 20:46:36.207862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:36.207845    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b54be71f-g6w4m"
Oct 30 20:46:36.218468 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:36.218438    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5chqb\" (UniqueName: \"kubernetes.io/projected/0b15da6e-2be7-4689-ae91-1f2b0910f69d-kube-api-access-5chqb\") pod \"maas-api-revoke-sub-cb0460da-vdhn6\" (UID: \"0b15da6e-2be7-4689-ae91-1f2b0910f69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb0460da-vdhn6"
Oct 30 20:46:36.218694 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:36.218663    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b15da6e-2be7-4689-ae91-1f2b0910f69d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb0460da-vdhn6\" (UID: \"0b15da6e-2be7-4689-ae91-1f2b0910f69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb0460da-vdhn6"
Oct 30 20:46:36.218941 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:36.218925    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b15da6e-2be7-4689-ae91-1f2b0910f69d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb0460da-vdhn6\" (UID: \"0b15da6e-2be7-4689-ae91-1f2b0910f69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb0460da-vdhn6"
Oct 30 20:46:36.265509 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:36.265468    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b54be71f-g6w4m"]
Oct 30 20:46:36.320235 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:36.320188    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b15da6e-2be7-4689-ae91-1f2b0910f69d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb0460da-vdhn6\" (UID: \"0b15da6e-2be7-4689-ae91-1f2b0910f69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb0460da-vdhn6"
Oct 30 20:46:36.320235 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:36.320260    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pkq7\" (UniqueName: \"kubernetes.io/projected/0951579d-a270-42f5-839a-7a555f465083-kube-api-access-2pkq7\") pod \"maas-api-revoke-sub-b54be71f-g6w4m\" (UID: \"0951579d-a270-42f5-839a-7a555f465083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b54be71f-g6w4m"
Oct 30 20:46:36.320523 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:36.320449    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0951579d-a270-42f5-839a-7a555f465083-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b54be71f-g6w4m\" (UID: \"0951579d-a270-42f5-839a-7a555f465083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b54be71f-g6w4m"
Oct 30 20:46:36.320523 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:36.320511    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5chqb\" (UniqueName: \"kubernetes.io/projected/0b15da6e-2be7-4689-ae91-1f2b0910f69d-kube-api-access-5chqb\") pod \"maas-api-revoke-sub-cb0460da-vdhn6\" (UID: \"0b15da6e-2be7-4689-ae91-1f2b0910f69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb0460da-vdhn6"
Oct 30 20:46:36.320645 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:36.320587    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0951579d-a270-42f5-839a-7a555f465083-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b54be71f-g6w4m\" (UID: \"0951579d-a270-42f5-839a-7a555f465083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b54be71f-g6w4m"
Oct 30 20:46:36.320741 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:36.320714    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b15da6e-2be7-4689-ae91-1f2b0910f69d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb0460da-vdhn6\" (UID: \"0b15da6e-2be7-4689-ae91-1f2b0910f69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb0460da-vdhn6"
Oct 30 20:46:36.321521 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:36.321493    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b15da6e-2be7-4689-ae91-1f2b0910f69d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb0460da-vdhn6\" (UID: \"0b15da6e-2be7-4689-ae91-1f2b0910f69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb0460da-vdhn6"
Oct 30 20:46:36.323047 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:36.323022    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b15da6e-2be7-4689-ae91-1f2b0910f69d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb0460da-vdhn6\" (UID: \"0b15da6e-2be7-4689-ae91-1f2b0910f69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb0460da-vdhn6"
Oct 30 20:46:36.379146 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:36.379076    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5chqb\" (UniqueName: \"kubernetes.io/projected/0b15da6e-2be7-4689-ae91-1f2b0910f69d-kube-api-access-5chqb\") pod \"maas-api-revoke-sub-cb0460da-vdhn6\" (UID: \"0b15da6e-2be7-4689-ae91-1f2b0910f69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb0460da-vdhn6"
Oct 30 20:46:36.401827 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:36.401803    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb0460da-vdhn6"
Oct 30 20:46:36.422397 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:36.422373    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0951579d-a270-42f5-839a-7a555f465083-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b54be71f-g6w4m\" (UID: \"0951579d-a270-42f5-839a-7a555f465083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b54be71f-g6w4m"
Oct 30 20:46:36.422557 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:36.422539    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pkq7\" (UniqueName: \"kubernetes.io/projected/0951579d-a270-42f5-839a-7a555f465083-kube-api-access-2pkq7\") pod \"maas-api-revoke-sub-b54be71f-g6w4m\" (UID: \"0951579d-a270-42f5-839a-7a555f465083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b54be71f-g6w4m"
Oct 30 20:46:36.423175 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:36.422683    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0951579d-a270-42f5-839a-7a555f465083-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b54be71f-g6w4m\" (UID: \"0951579d-a270-42f5-839a-7a555f465083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b54be71f-g6w4m"
Oct 30 20:46:36.423288 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:36.423256    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0951579d-a270-42f5-839a-7a555f465083-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b54be71f-g6w4m\" (UID: \"0951579d-a270-42f5-839a-7a555f465083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b54be71f-g6w4m"
Oct 30 20:46:36.425388 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:36.425360    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0951579d-a270-42f5-839a-7a555f465083-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b54be71f-g6w4m\" (UID: \"0951579d-a270-42f5-839a-7a555f465083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b54be71f-g6w4m"
Oct 30 20:46:36.485056 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:36.485012    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pkq7\" (UniqueName: \"kubernetes.io/projected/0951579d-a270-42f5-839a-7a555f465083-kube-api-access-2pkq7\") pod \"maas-api-revoke-sub-b54be71f-g6w4m\" (UID: \"0951579d-a270-42f5-839a-7a555f465083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b54be71f-g6w4m"
Oct 30 20:46:36.526041 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:36.526011    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b54be71f-g6w4m"
Oct 30 20:46:36.785932 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:46:36.785896    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b15da6e_2be7_4689_ae91_1f2b0910f69d.slice/crio-e564ff18cb7fd3e926b307b4f64db6581a3f02e2c8bd5cd16ebc42e62817e882 WatchSource:0}: Error finding container e564ff18cb7fd3e926b307b4f64db6581a3f02e2c8bd5cd16ebc42e62817e882: Status 404 returned error can't find the container with id e564ff18cb7fd3e926b307b4f64db6581a3f02e2c8bd5cd16ebc42e62817e882
Oct 30 20:46:36.786978 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:36.786959    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb0460da-vdhn6"]
Oct 30 20:46:36.915800 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:46:36.915772    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0951579d_a270_42f5_839a_7a555f465083.slice/crio-c2f294b58176e0905157ba7c508f3fd84ae947fb3ea7609d9277ee4e0ae70599 WatchSource:0}: Error finding container c2f294b58176e0905157ba7c508f3fd84ae947fb3ea7609d9277ee4e0ae70599: Status 404 returned error can't find the container with id c2f294b58176e0905157ba7c508f3fd84ae947fb3ea7609d9277ee4e0ae70599
Oct 30 20:46:36.916296 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:36.916270    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b54be71f-g6w4m"]
Oct 30 20:46:37.796940 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:37.796908    5071 generic.go:356] "Generic (PLEG): container finished" podID="0b15da6e-2be7-4689-ae91-1f2b0910f69d" containerID="0ce2f6c6426d841e667dcba8348b3280a07d98db30bf06555f8f345c1f7deb82" exitCode=0
Oct 30 20:46:37.797356 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:37.796996    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb0460da-vdhn6" event={"ID":"0b15da6e-2be7-4689-ae91-1f2b0910f69d","Type":"ContainerDied","Data":"0ce2f6c6426d841e667dcba8348b3280a07d98db30bf06555f8f345c1f7deb82"}
Oct 30 20:46:37.797356 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:37.797048    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb0460da-vdhn6" event={"ID":"0b15da6e-2be7-4689-ae91-1f2b0910f69d","Type":"ContainerStarted","Data":"e564ff18cb7fd3e926b307b4f64db6581a3f02e2c8bd5cd16ebc42e62817e882"}
Oct 30 20:46:37.798862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:37.798806    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b54be71f-g6w4m" event={"ID":"0951579d-a270-42f5-839a-7a555f465083","Type":"ContainerStarted","Data":"a4092b4b27794dea1811f53e63625ed1506201759dc5b277f6af9a6bb5df6634"}
Oct 30 20:46:37.798978 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:37.798879    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b54be71f-g6w4m" event={"ID":"0951579d-a270-42f5-839a-7a555f465083","Type":"ContainerStarted","Data":"c2f294b58176e0905157ba7c508f3fd84ae947fb3ea7609d9277ee4e0ae70599"}
Oct 30 20:46:38.030735 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:38.030554    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b54be71f-g6w4m" podStartSLOduration=2.030533289 podStartE2EDuration="2.030533289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:38.02527878 +0000 UTC m=+1209.942300316" watchObservedRunningTime="2026-10-30 20:46:38.030533289 +0000 UTC m=+1209.947554826"
Oct 30 20:46:38.774412 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:38.774374    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:38.807921 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:38.807888    5071 generic.go:356] "Generic (PLEG): container finished" podID="0951579d-a270-42f5-839a-7a555f465083" containerID="a4092b4b27794dea1811f53e63625ed1506201759dc5b277f6af9a6bb5df6634" exitCode=0
Oct 30 20:46:38.808393 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:38.807969    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b54be71f-g6w4m" event={"ID":"0951579d-a270-42f5-839a-7a555f465083","Type":"ContainerDied","Data":"a4092b4b27794dea1811f53e63625ed1506201759dc5b277f6af9a6bb5df6634"}
Oct 30 20:46:39.000300 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:39.000277    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb0460da-vdhn6"
Oct 30 20:46:39.069857 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:39.069809    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b15da6e-2be7-4689-ae91-1f2b0910f69d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b15da6e-2be7-4689-ae91-1f2b0910f69d-maas-api-openshift-ca\") pod \"0b15da6e-2be7-4689-ae91-1f2b0910f69d\" (UID: \"0b15da6e-2be7-4689-ae91-1f2b0910f69d\") "
Oct 30 20:46:39.070043 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:39.069990    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b15da6e-2be7-4689-ae91-1f2b0910f69d-kube-api-access-5chqb\" (UniqueName: \"kubernetes.io/projected/0b15da6e-2be7-4689-ae91-1f2b0910f69d-kube-api-access-5chqb\") pod \"0b15da6e-2be7-4689-ae91-1f2b0910f69d\" (UID: \"0b15da6e-2be7-4689-ae91-1f2b0910f69d\") "
Oct 30 20:46:39.070166 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:39.070147    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b15da6e-2be7-4689-ae91-1f2b0910f69d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b15da6e-2be7-4689-ae91-1f2b0910f69d-maas-api-xks-ca\") pod \"0b15da6e-2be7-4689-ae91-1f2b0910f69d\" (UID: \"0b15da6e-2be7-4689-ae91-1f2b0910f69d\") "
Oct 30 20:46:39.070243 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:39.070179    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b15da6e-2be7-4689-ae91-1f2b0910f69d-maas-api-openshift-ca" pod "0b15da6e-2be7-4689-ae91-1f2b0910f69d" (UID: "0b15da6e-2be7-4689-ae91-1f2b0910f69d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:46:39.070789 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:39.070718    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b15da6e-2be7-4689-ae91-1f2b0910f69d-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:39.072362 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:39.072322    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b15da6e-2be7-4689-ae91-1f2b0910f69d-kube-api-access-5chqb" pod "0b15da6e-2be7-4689-ae91-1f2b0910f69d" (UID: "0b15da6e-2be7-4689-ae91-1f2b0910f69d"). InnerVolumeSpecName "kube-api-access-5chqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:46:39.072477 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:39.072433    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b15da6e-2be7-4689-ae91-1f2b0910f69d-maas-api-xks-ca" pod "0b15da6e-2be7-4689-ae91-1f2b0910f69d" (UID: "0b15da6e-2be7-4689-ae91-1f2b0910f69d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:46:39.172154 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:39.172114    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5chqb\" (UniqueName: \"kubernetes.io/projected/0b15da6e-2be7-4689-ae91-1f2b0910f69d-kube-api-access-5chqb\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:39.172154 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:39.172159    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b15da6e-2be7-4689-ae91-1f2b0910f69d-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:39.816371 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:39.816338    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb0460da-vdhn6"
Oct 30 20:46:39.816815 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:39.816373    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb0460da-vdhn6" event={"ID":"0b15da6e-2be7-4689-ae91-1f2b0910f69d","Type":"ContainerDied","Data":"e564ff18cb7fd3e926b307b4f64db6581a3f02e2c8bd5cd16ebc42e62817e882"}
Oct 30 20:46:39.816815 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:39.816432    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e564ff18cb7fd3e926b307b4f64db6581a3f02e2c8bd5cd16ebc42e62817e882"
Oct 30 20:46:40.008869 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:40.008846    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b54be71f-g6w4m"
Oct 30 20:46:40.085514 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:40.085427    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0951579d-a270-42f5-839a-7a555f465083-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0951579d-a270-42f5-839a-7a555f465083-maas-api-openshift-ca\") pod \"0951579d-a270-42f5-839a-7a555f465083\" (UID: \"0951579d-a270-42f5-839a-7a555f465083\") "
Oct 30 20:46:40.085697 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:40.085517    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0951579d-a270-42f5-839a-7a555f465083-kube-api-access-2pkq7\" (UniqueName: \"kubernetes.io/projected/0951579d-a270-42f5-839a-7a555f465083-kube-api-access-2pkq7\") pod \"0951579d-a270-42f5-839a-7a555f465083\" (UID: \"0951579d-a270-42f5-839a-7a555f465083\") "
Oct 30 20:46:40.085697 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:40.085555    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0951579d-a270-42f5-839a-7a555f465083-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0951579d-a270-42f5-839a-7a555f465083-maas-api-xks-ca\") pod \"0951579d-a270-42f5-839a-7a555f465083\" (UID: \"0951579d-a270-42f5-839a-7a555f465083\") "
Oct 30 20:46:40.085901 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:40.085878    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0951579d-a270-42f5-839a-7a555f465083-maas-api-openshift-ca" pod "0951579d-a270-42f5-839a-7a555f465083" (UID: "0951579d-a270-42f5-839a-7a555f465083"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:46:40.086173 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:40.086151    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0951579d-a270-42f5-839a-7a555f465083-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:40.087836 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:40.087801    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0951579d-a270-42f5-839a-7a555f465083-kube-api-access-2pkq7" pod "0951579d-a270-42f5-839a-7a555f465083" (UID: "0951579d-a270-42f5-839a-7a555f465083"). InnerVolumeSpecName "kube-api-access-2pkq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:46:40.087934 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:40.087859    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0951579d-a270-42f5-839a-7a555f465083-maas-api-xks-ca" pod "0951579d-a270-42f5-839a-7a555f465083" (UID: "0951579d-a270-42f5-839a-7a555f465083"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:46:40.187673 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:40.187643    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pkq7\" (UniqueName: \"kubernetes.io/projected/0951579d-a270-42f5-839a-7a555f465083-kube-api-access-2pkq7\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:40.187673 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:40.187671    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0951579d-a270-42f5-839a-7a555f465083-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:40.825766 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:40.825724    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b54be71f-g6w4m" event={"ID":"0951579d-a270-42f5-839a-7a555f465083","Type":"ContainerDied","Data":"c2f294b58176e0905157ba7c508f3fd84ae947fb3ea7609d9277ee4e0ae70599"}
Oct 30 20:46:40.825766 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:40.825756    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b54be71f-g6w4m"
Oct 30 20:46:40.826236 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:40.825789    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2f294b58176e0905157ba7c508f3fd84ae947fb3ea7609d9277ee4e0ae70599"
Oct 30 20:46:41.060743 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:41.060706    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-640e29ac-57vxk"]
Oct 30 20:46:41.062748 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:41.062716    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-640e29ac-57vxk"]
Oct 30 20:46:42.203011 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:42.202978    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6280eea7-d203-4c65-883d-a8027a1984da" path="/var/lib/kubelet/pods/6280eea7-d203-4c65-883d-a8027a1984da/volumes"
Oct 30 20:46:43.733864 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:43.731456    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:46.735528 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:46:46.734006    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:46:46.869209 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:46.869172    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44d167d3-9p9hx"]
Oct 30 20:46:46.870092 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:46.870071    5071 cpu_manager.go:414] "containerMap: removing container" podUID="0b15da6e-2be7-4689-ae91-1f2b0910f69d" containerName="revoke-keys"
Oct 30 20:46:46.870170 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:46.870095    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b15da6e-2be7-4689-ae91-1f2b0910f69d" containerName="revoke-keys"
Oct 30 20:46:46.870170 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:46.870163    5071 cpu_manager.go:414] "containerMap: removing container" podUID="0951579d-a270-42f5-839a-7a555f465083" containerName="revoke-keys"
Oct 30 20:46:46.870258 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:46.870170    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0951579d-a270-42f5-839a-7a555f465083" containerName="revoke-keys"
Oct 30 20:46:46.870339 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:46.870326    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="0951579d-a270-42f5-839a-7a555f465083" containerName="revoke-keys"
Oct 30 20:46:46.870388 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:46.870347    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b15da6e-2be7-4689-ae91-1f2b0910f69d" containerName="revoke-keys"
Oct 30 20:46:46.874524 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:46.874489    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d167d3-9p9hx"
Oct 30 20:46:46.930785 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:46.930756    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44d167d3-9p9hx"]
Oct 30 20:46:46.974337 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:46.974305    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:46:46.975356 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:46.975322    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc73f251-c609-4a35-9757-203c3595c1a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44d167d3-9p9hx\" (UID: \"cc73f251-c609-4a35-9757-203c3595c1a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d167d3-9p9hx"
Oct 30 20:46:46.975535 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:46.975505    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc73f251-c609-4a35-9757-203c3595c1a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44d167d3-9p9hx\" (UID: \"cc73f251-c609-4a35-9757-203c3595c1a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d167d3-9p9hx"
Oct 30 20:46:46.975535 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:46.975516    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 20:46:46.975662 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:46.975591    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlsfc\" (UniqueName: \"kubernetes.io/projected/cc73f251-c609-4a35-9757-203c3595c1a8-kube-api-access-zlsfc\") pod \"maas-api-revoke-sub-44d167d3-9p9hx\" (UID: \"cc73f251-c609-4a35-9757-203c3595c1a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d167d3-9p9hx"
Oct 30 20:46:47.077207 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:47.077170    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc73f251-c609-4a35-9757-203c3595c1a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44d167d3-9p9hx\" (UID: \"cc73f251-c609-4a35-9757-203c3595c1a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d167d3-9p9hx"
Oct 30 20:46:47.077400 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:47.077280    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc73f251-c609-4a35-9757-203c3595c1a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44d167d3-9p9hx\" (UID: \"cc73f251-c609-4a35-9757-203c3595c1a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d167d3-9p9hx"
Oct 30 20:46:47.077400 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:47.077325    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlsfc\" (UniqueName: \"kubernetes.io/projected/cc73f251-c609-4a35-9757-203c3595c1a8-kube-api-access-zlsfc\") pod \"maas-api-revoke-sub-44d167d3-9p9hx\" (UID: \"cc73f251-c609-4a35-9757-203c3595c1a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d167d3-9p9hx"
Oct 30 20:46:47.077790 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:47.077767    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc73f251-c609-4a35-9757-203c3595c1a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44d167d3-9p9hx\" (UID: \"cc73f251-c609-4a35-9757-203c3595c1a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d167d3-9p9hx"
Oct 30 20:46:47.079868 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:47.079840    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc73f251-c609-4a35-9757-203c3595c1a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44d167d3-9p9hx\" (UID: \"cc73f251-c609-4a35-9757-203c3595c1a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d167d3-9p9hx"
Oct 30 20:46:47.137750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:47.137722    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlsfc\" (UniqueName: \"kubernetes.io/projected/cc73f251-c609-4a35-9757-203c3595c1a8-kube-api-access-zlsfc\") pod \"maas-api-revoke-sub-44d167d3-9p9hx\" (UID: \"cc73f251-c609-4a35-9757-203c3595c1a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d167d3-9p9hx"
Oct 30 20:46:47.190046 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:47.190014    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d167d3-9p9hx"
Oct 30 20:46:47.588616 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:47.588571    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44d167d3-9p9hx"]
Oct 30 20:46:47.588744 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:46:47.588578    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc73f251_c609_4a35_9757_203c3595c1a8.slice/crio-ffb028ee49c1a994401126ad90481899f1d5e55f714a2f9ca01a5bf83aefb676 WatchSource:0}: Error finding container ffb028ee49c1a994401126ad90481899f1d5e55f714a2f9ca01a5bf83aefb676: Status 404 returned error can't find the container with id ffb028ee49c1a994401126ad90481899f1d5e55f714a2f9ca01a5bf83aefb676
Oct 30 20:46:47.800068 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:47.800037    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfd785c6-h5zw5"]
Oct 30 20:46:47.805586 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:47.805566    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd785c6-h5zw5"
Oct 30 20:46:47.863041 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:47.863018    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfd785c6-h5zw5"]
Oct 30 20:46:47.885741 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:47.885711    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d167d3-9p9hx" event={"ID":"cc73f251-c609-4a35-9757-203c3595c1a8","Type":"ContainerStarted","Data":"acd97f9211a912240d1a112f3b74fd09e99b5eb88a7b9cf7e867fc6db7714d9b"}
Oct 30 20:46:47.885906 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:47.885763    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d167d3-9p9hx" event={"ID":"cc73f251-c609-4a35-9757-203c3595c1a8","Type":"ContainerStarted","Data":"ffb028ee49c1a994401126ad90481899f1d5e55f714a2f9ca01a5bf83aefb676"}
Oct 30 20:46:47.990555 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:47.990506    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/073e76de-dacd-49a5-b054-a19c929c0491-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfd785c6-h5zw5\" (UID: \"073e76de-dacd-49a5-b054-a19c929c0491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd785c6-h5zw5"
Oct 30 20:46:47.990555 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:47.990552    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phbt5\" (UniqueName: \"kubernetes.io/projected/073e76de-dacd-49a5-b054-a19c929c0491-kube-api-access-phbt5\") pod \"maas-api-revoke-sub-cfd785c6-h5zw5\" (UID: \"073e76de-dacd-49a5-b054-a19c929c0491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd785c6-h5zw5"
Oct 30 20:46:47.990841 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:47.990612    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/073e76de-dacd-49a5-b054-a19c929c0491-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfd785c6-h5zw5\" (UID: \"073e76de-dacd-49a5-b054-a19c929c0491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd785c6-h5zw5"
Oct 30 20:46:48.038649 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:48.038600    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d167d3-9p9hx" podStartSLOduration=2.038587525 podStartE2EDuration="2.038587525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:48.03537327 +0000 UTC m=+1219.952394806" watchObservedRunningTime="2026-10-30 20:46:48.038587525 +0000 UTC m=+1219.955609061"
Oct 30 20:46:48.092301 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:48.092212    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/073e76de-dacd-49a5-b054-a19c929c0491-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfd785c6-h5zw5\" (UID: \"073e76de-dacd-49a5-b054-a19c929c0491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd785c6-h5zw5"
Oct 30 20:46:48.092301 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:48.092246    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phbt5\" (UniqueName: \"kubernetes.io/projected/073e76de-dacd-49a5-b054-a19c929c0491-kube-api-access-phbt5\") pod \"maas-api-revoke-sub-cfd785c6-h5zw5\" (UID: \"073e76de-dacd-49a5-b054-a19c929c0491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd785c6-h5zw5"
Oct 30 20:46:48.092301 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:48.092276    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/073e76de-dacd-49a5-b054-a19c929c0491-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfd785c6-h5zw5\" (UID: \"073e76de-dacd-49a5-b054-a19c929c0491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd785c6-h5zw5"
Oct 30 20:46:48.093172 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:48.093145    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/073e76de-dacd-49a5-b054-a19c929c0491-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfd785c6-h5zw5\" (UID: \"073e76de-dacd-49a5-b054-a19c929c0491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd785c6-h5zw5"
Oct 30 20:46:48.095020 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:48.095000    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/073e76de-dacd-49a5-b054-a19c929c0491-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfd785c6-h5zw5\" (UID: \"073e76de-dacd-49a5-b054-a19c929c0491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd785c6-h5zw5"
Oct 30 20:46:48.151641 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:48.151606    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phbt5\" (UniqueName: \"kubernetes.io/projected/073e76de-dacd-49a5-b054-a19c929c0491-kube-api-access-phbt5\") pod \"maas-api-revoke-sub-cfd785c6-h5zw5\" (UID: \"073e76de-dacd-49a5-b054-a19c929c0491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd785c6-h5zw5"
Oct 30 20:46:48.419766 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:48.419677    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd785c6-h5zw5"
Oct 30 20:46:48.618427 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:48.618364    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:48.825165 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:46:48.825124    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod073e76de_dacd_49a5_b054_a19c929c0491.slice/crio-2b2015e6f98f5a1294e9dc7d704f3c81132ce9276c1f93e2ca7b0fd107ce85f5 WatchSource:0}: Error finding container 2b2015e6f98f5a1294e9dc7d704f3c81132ce9276c1f93e2ca7b0fd107ce85f5: Status 404 returned error can't find the container with id 2b2015e6f98f5a1294e9dc7d704f3c81132ce9276c1f93e2ca7b0fd107ce85f5
Oct 30 20:46:48.826812 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:48.826784    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfd785c6-h5zw5"]
Oct 30 20:46:48.894727 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:48.894697    5071 generic.go:356] "Generic (PLEG): container finished" podID="cc73f251-c609-4a35-9757-203c3595c1a8" containerID="acd97f9211a912240d1a112f3b74fd09e99b5eb88a7b9cf7e867fc6db7714d9b" exitCode=0
Oct 30 20:46:48.894914 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:48.894781    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d167d3-9p9hx" event={"ID":"cc73f251-c609-4a35-9757-203c3595c1a8","Type":"ContainerDied","Data":"acd97f9211a912240d1a112f3b74fd09e99b5eb88a7b9cf7e867fc6db7714d9b"}
Oct 30 20:46:48.896346 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:48.896323    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd785c6-h5zw5" event={"ID":"073e76de-dacd-49a5-b054-a19c929c0491","Type":"ContainerStarted","Data":"2b2015e6f98f5a1294e9dc7d704f3c81132ce9276c1f93e2ca7b0fd107ce85f5"}
Oct 30 20:46:49.907006 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:49.906970    5071 generic.go:356] "Generic (PLEG): container finished" podID="073e76de-dacd-49a5-b054-a19c929c0491" containerID="53ef8f47c3b223fd57e3f20d6b41237537d18d596752efd5042718c9ae17f1e3" exitCode=0
Oct 30 20:46:49.907420 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:49.907044    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd785c6-h5zw5" event={"ID":"073e76de-dacd-49a5-b054-a19c929c0491","Type":"ContainerDied","Data":"53ef8f47c3b223fd57e3f20d6b41237537d18d596752efd5042718c9ae17f1e3"}
Oct 30 20:46:50.116209 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:50.116181    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d167d3-9p9hx"
Oct 30 20:46:50.128626 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:50.128602    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc73f251-c609-4a35-9757-203c3595c1a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc73f251-c609-4a35-9757-203c3595c1a8-maas-api-openshift-ca\") pod \"cc73f251-c609-4a35-9757-203c3595c1a8\" (UID: \"cc73f251-c609-4a35-9757-203c3595c1a8\") "
Oct 30 20:46:50.129157 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:50.128807    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc73f251-c609-4a35-9757-203c3595c1a8-kube-api-access-zlsfc\" (UniqueName: \"kubernetes.io/projected/cc73f251-c609-4a35-9757-203c3595c1a8-kube-api-access-zlsfc\") pod \"cc73f251-c609-4a35-9757-203c3595c1a8\" (UID: \"cc73f251-c609-4a35-9757-203c3595c1a8\") "
Oct 30 20:46:50.129157 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:50.128975    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc73f251-c609-4a35-9757-203c3595c1a8-maas-api-openshift-ca" pod "cc73f251-c609-4a35-9757-203c3595c1a8" (UID: "cc73f251-c609-4a35-9757-203c3595c1a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:46:50.129157 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:50.129109    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc73f251-c609-4a35-9757-203c3595c1a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc73f251-c609-4a35-9757-203c3595c1a8-maas-api-xks-ca\") pod \"cc73f251-c609-4a35-9757-203c3595c1a8\" (UID: \"cc73f251-c609-4a35-9757-203c3595c1a8\") "
Oct 30 20:46:50.129812 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:50.129790    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc73f251-c609-4a35-9757-203c3595c1a8-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:50.131237 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:50.131214    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc73f251-c609-4a35-9757-203c3595c1a8-kube-api-access-zlsfc" pod "cc73f251-c609-4a35-9757-203c3595c1a8" (UID: "cc73f251-c609-4a35-9757-203c3595c1a8"). InnerVolumeSpecName "kube-api-access-zlsfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:46:50.131535 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:50.131507    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc73f251-c609-4a35-9757-203c3595c1a8-maas-api-xks-ca" pod "cc73f251-c609-4a35-9757-203c3595c1a8" (UID: "cc73f251-c609-4a35-9757-203c3595c1a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:46:50.231418 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:50.231372    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlsfc\" (UniqueName: \"kubernetes.io/projected/cc73f251-c609-4a35-9757-203c3595c1a8-kube-api-access-zlsfc\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:50.231418 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:50.231406    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc73f251-c609-4a35-9757-203c3595c1a8-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:50.917396 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:50.917360    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d167d3-9p9hx"
Oct 30 20:46:50.917859 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:50.917382    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d167d3-9p9hx" event={"ID":"cc73f251-c609-4a35-9757-203c3595c1a8","Type":"ContainerDied","Data":"ffb028ee49c1a994401126ad90481899f1d5e55f714a2f9ca01a5bf83aefb676"}
Oct 30 20:46:50.917859 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:50.917456    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffb028ee49c1a994401126ad90481899f1d5e55f714a2f9ca01a5bf83aefb676"
Oct 30 20:46:51.089025 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:51.089000    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd785c6-h5zw5"
Oct 30 20:46:51.141903 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:51.141870    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/073e76de-dacd-49a5-b054-a19c929c0491-kube-api-access-phbt5\" (UniqueName: \"kubernetes.io/projected/073e76de-dacd-49a5-b054-a19c929c0491-kube-api-access-phbt5\") pod \"073e76de-dacd-49a5-b054-a19c929c0491\" (UID: \"073e76de-dacd-49a5-b054-a19c929c0491\") "
Oct 30 20:46:51.142115 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:51.142026    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/073e76de-dacd-49a5-b054-a19c929c0491-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/073e76de-dacd-49a5-b054-a19c929c0491-maas-api-xks-ca\") pod \"073e76de-dacd-49a5-b054-a19c929c0491\" (UID: \"073e76de-dacd-49a5-b054-a19c929c0491\") "
Oct 30 20:46:51.142115 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:51.142061    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/073e76de-dacd-49a5-b054-a19c929c0491-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/073e76de-dacd-49a5-b054-a19c929c0491-maas-api-openshift-ca\") pod \"073e76de-dacd-49a5-b054-a19c929c0491\" (UID: \"073e76de-dacd-49a5-b054-a19c929c0491\") "
Oct 30 20:46:51.142538 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:51.142507    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/073e76de-dacd-49a5-b054-a19c929c0491-maas-api-openshift-ca" pod "073e76de-dacd-49a5-b054-a19c929c0491" (UID: "073e76de-dacd-49a5-b054-a19c929c0491"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:46:51.142714 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:51.142686    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/073e76de-dacd-49a5-b054-a19c929c0491-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:51.144772 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:51.144746    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/073e76de-dacd-49a5-b054-a19c929c0491-kube-api-access-phbt5" pod "073e76de-dacd-49a5-b054-a19c929c0491" (UID: "073e76de-dacd-49a5-b054-a19c929c0491"). InnerVolumeSpecName "kube-api-access-phbt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:46:51.144904 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:51.144814    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/073e76de-dacd-49a5-b054-a19c929c0491-maas-api-xks-ca" pod "073e76de-dacd-49a5-b054-a19c929c0491" (UID: "073e76de-dacd-49a5-b054-a19c929c0491"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:46:51.244310 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:51.244204    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/073e76de-dacd-49a5-b054-a19c929c0491-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:51.244310 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:51.244234    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phbt5\" (UniqueName: \"kubernetes.io/projected/073e76de-dacd-49a5-b054-a19c929c0491-kube-api-access-phbt5\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:51.694229 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:51.694188    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b447559-rmqt4"]
Oct 30 20:46:51.694894 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:51.694876    5071 cpu_manager.go:414] "containerMap: removing container" podUID="cc73f251-c609-4a35-9757-203c3595c1a8" containerName="revoke-keys"
Oct 30 20:46:51.694894 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:51.694892    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc73f251-c609-4a35-9757-203c3595c1a8" containerName="revoke-keys"
Oct 30 20:46:51.695036 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:51.694902    5071 cpu_manager.go:414] "containerMap: removing container" podUID="073e76de-dacd-49a5-b054-a19c929c0491" containerName="revoke-keys"
Oct 30 20:46:51.695036 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:51.694908    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="073e76de-dacd-49a5-b054-a19c929c0491" containerName="revoke-keys"
Oct 30 20:46:51.695036 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:51.695031    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc73f251-c609-4a35-9757-203c3595c1a8" containerName="revoke-keys"
Oct 30 20:46:51.695140 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:51.695039    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="073e76de-dacd-49a5-b054-a19c929c0491" containerName="revoke-keys"
Oct 30 20:46:51.698889 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:51.698872    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b447559-rmqt4"
Oct 30 20:46:51.751256 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:51.751226    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5de98b75-5345-48cd-8b72-31d9f5a149b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b447559-rmqt4\" (UID: \"5de98b75-5345-48cd-8b72-31d9f5a149b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b447559-rmqt4"
Oct 30 20:46:51.751447 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:51.751283    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4wbm\" (UniqueName: \"kubernetes.io/projected/5de98b75-5345-48cd-8b72-31d9f5a149b8-kube-api-access-z4wbm\") pod \"maas-api-revoke-sub-3b447559-rmqt4\" (UID: \"5de98b75-5345-48cd-8b72-31d9f5a149b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b447559-rmqt4"
Oct 30 20:46:51.751615 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:51.751578    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5de98b75-5345-48cd-8b72-31d9f5a149b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b447559-rmqt4\" (UID: \"5de98b75-5345-48cd-8b72-31d9f5a149b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b447559-rmqt4"
Oct 30 20:46:51.758344 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:51.758309    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b447559-rmqt4"]
Oct 30 20:46:51.853376 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:51.853343    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5de98b75-5345-48cd-8b72-31d9f5a149b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b447559-rmqt4\" (UID: \"5de98b75-5345-48cd-8b72-31d9f5a149b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b447559-rmqt4"
Oct 30 20:46:51.853525 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:51.853449    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5de98b75-5345-48cd-8b72-31d9f5a149b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b447559-rmqt4\" (UID: \"5de98b75-5345-48cd-8b72-31d9f5a149b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b447559-rmqt4"
Oct 30 20:46:51.853606 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:51.853525    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4wbm\" (UniqueName: \"kubernetes.io/projected/5de98b75-5345-48cd-8b72-31d9f5a149b8-kube-api-access-z4wbm\") pod \"maas-api-revoke-sub-3b447559-rmqt4\" (UID: \"5de98b75-5345-48cd-8b72-31d9f5a149b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b447559-rmqt4"
Oct 30 20:46:51.854225 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:51.854148    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5de98b75-5345-48cd-8b72-31d9f5a149b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b447559-rmqt4\" (UID: \"5de98b75-5345-48cd-8b72-31d9f5a149b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b447559-rmqt4"
Oct 30 20:46:51.856001 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:51.855980    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5de98b75-5345-48cd-8b72-31d9f5a149b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b447559-rmqt4\" (UID: \"5de98b75-5345-48cd-8b72-31d9f5a149b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b447559-rmqt4"
Oct 30 20:46:51.913274 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:51.913249    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4wbm\" (UniqueName: \"kubernetes.io/projected/5de98b75-5345-48cd-8b72-31d9f5a149b8-kube-api-access-z4wbm\") pod \"maas-api-revoke-sub-3b447559-rmqt4\" (UID: \"5de98b75-5345-48cd-8b72-31d9f5a149b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b447559-rmqt4"
Oct 30 20:46:51.927537 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:51.927513    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd785c6-h5zw5"
Oct 30 20:46:51.927893 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:51.927539    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd785c6-h5zw5" event={"ID":"073e76de-dacd-49a5-b054-a19c929c0491","Type":"ContainerDied","Data":"2b2015e6f98f5a1294e9dc7d704f3c81132ce9276c1f93e2ca7b0fd107ce85f5"}
Oct 30 20:46:51.927893 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:51.927586    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b2015e6f98f5a1294e9dc7d704f3c81132ce9276c1f93e2ca7b0fd107ce85f5"
Oct 30 20:46:52.013323 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:52.013249    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b447559-rmqt4"
Oct 30 20:46:52.400688 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:46:52.400654    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5de98b75_5345_48cd_8b72_31d9f5a149b8.slice/crio-6355f79dc74f85c15fc736d6c59da3e9c83488337e3659d2065f2d640617e3c2 WatchSource:0}: Error finding container 6355f79dc74f85c15fc736d6c59da3e9c83488337e3659d2065f2d640617e3c2: Status 404 returned error can't find the container with id 6355f79dc74f85c15fc736d6c59da3e9c83488337e3659d2065f2d640617e3c2
Oct 30 20:46:52.401631 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:52.401608    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b447559-rmqt4"]
Oct 30 20:46:52.938936 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:52.938901    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b447559-rmqt4" event={"ID":"5de98b75-5345-48cd-8b72-31d9f5a149b8","Type":"ContainerStarted","Data":"c85eeac275a3bd5ac21b137dc48f73f726140ebb8b51c2c2ad18c2a21ac99af3"}
Oct 30 20:46:52.939354 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:52.938956    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b447559-rmqt4" event={"ID":"5de98b75-5345-48cd-8b72-31d9f5a149b8","Type":"ContainerStarted","Data":"6355f79dc74f85c15fc736d6c59da3e9c83488337e3659d2065f2d640617e3c2"}
Oct 30 20:46:53.053968 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:53.053934    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdb77bd8-x6tld"]
Oct 30 20:46:53.058204 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:53.058176    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdb77bd8-x6tld"]
Oct 30 20:46:53.100490 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:53.100440    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b447559-rmqt4" podStartSLOduration=2.100425645 podStartE2EDuration="2.100425645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:53.098933823 +0000 UTC m=+1225.015955358" watchObservedRunningTime="2026-10-30 20:46:53.100425645 +0000 UTC m=+1225.017447190"
Oct 30 20:46:53.581943 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:53.581910    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:46:53.948941 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:53.948904    5071 generic.go:356] "Generic (PLEG): container finished" podID="5de98b75-5345-48cd-8b72-31d9f5a149b8" containerID="c85eeac275a3bd5ac21b137dc48f73f726140ebb8b51c2c2ad18c2a21ac99af3" exitCode=0
Oct 30 20:46:53.949366 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:53.948989    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b447559-rmqt4" event={"ID":"5de98b75-5345-48cd-8b72-31d9f5a149b8","Type":"ContainerDied","Data":"c85eeac275a3bd5ac21b137dc48f73f726140ebb8b51c2c2ad18c2a21ac99af3"}
Oct 30 20:46:54.204924 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:54.204814    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f376fda3-4de7-4c54-a64a-e2af38ecb340" path="/var/lib/kubelet/pods/f376fda3-4de7-4c54-a64a-e2af38ecb340/volumes"
Oct 30 20:46:55.132160 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:55.132128    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b447559-rmqt4"
Oct 30 20:46:55.201071 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:55.201039    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5de98b75-5345-48cd-8b72-31d9f5a149b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5de98b75-5345-48cd-8b72-31d9f5a149b8-maas-api-openshift-ca\") pod \"5de98b75-5345-48cd-8b72-31d9f5a149b8\" (UID: \"5de98b75-5345-48cd-8b72-31d9f5a149b8\") "
Oct 30 20:46:55.201218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:55.201137    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5de98b75-5345-48cd-8b72-31d9f5a149b8-kube-api-access-z4wbm\" (UniqueName: \"kubernetes.io/projected/5de98b75-5345-48cd-8b72-31d9f5a149b8-kube-api-access-z4wbm\") pod \"5de98b75-5345-48cd-8b72-31d9f5a149b8\" (UID: \"5de98b75-5345-48cd-8b72-31d9f5a149b8\") "
Oct 30 20:46:55.201218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:55.201193    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5de98b75-5345-48cd-8b72-31d9f5a149b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5de98b75-5345-48cd-8b72-31d9f5a149b8-maas-api-xks-ca\") pod \"5de98b75-5345-48cd-8b72-31d9f5a149b8\" (UID: \"5de98b75-5345-48cd-8b72-31d9f5a149b8\") "
Oct 30 20:46:55.201403 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:55.201381    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5de98b75-5345-48cd-8b72-31d9f5a149b8-maas-api-openshift-ca" pod "5de98b75-5345-48cd-8b72-31d9f5a149b8" (UID: "5de98b75-5345-48cd-8b72-31d9f5a149b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:46:55.201664 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:55.201647    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5de98b75-5345-48cd-8b72-31d9f5a149b8-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:55.203369 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:55.203337    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5de98b75-5345-48cd-8b72-31d9f5a149b8-maas-api-xks-ca" pod "5de98b75-5345-48cd-8b72-31d9f5a149b8" (UID: "5de98b75-5345-48cd-8b72-31d9f5a149b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:46:55.203472 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:55.203433    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5de98b75-5345-48cd-8b72-31d9f5a149b8-kube-api-access-z4wbm" pod "5de98b75-5345-48cd-8b72-31d9f5a149b8" (UID: "5de98b75-5345-48cd-8b72-31d9f5a149b8"). InnerVolumeSpecName "kube-api-access-z4wbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:46:55.303901 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:55.303802    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4wbm\" (UniqueName: \"kubernetes.io/projected/5de98b75-5345-48cd-8b72-31d9f5a149b8-kube-api-access-z4wbm\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:55.303901 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:55.303862    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5de98b75-5345-48cd-8b72-31d9f5a149b8-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:55.966718 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:55.966676    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b447559-rmqt4" event={"ID":"5de98b75-5345-48cd-8b72-31d9f5a149b8","Type":"ContainerDied","Data":"6355f79dc74f85c15fc736d6c59da3e9c83488337e3659d2065f2d640617e3c2"}
Oct 30 20:46:55.966718 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:55.966701    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b447559-rmqt4"
Oct 30 20:46:55.966948 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:55.966772    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6355f79dc74f85c15fc736d6c59da3e9c83488337e3659d2065f2d640617e3c2"
Oct 30 20:46:57.064266 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:57.064230    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e580a5-ftxs5"]
Oct 30 20:46:57.111210 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:57.111172    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e580a5-ftxs5"]
Oct 30 20:46:58.203471 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:58.203439    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e936350a-b13c-45e6-ac5a-8d2097b9bba0" path="/var/lib/kubelet/pods/e936350a-b13c-45e6-ac5a-8d2097b9bba0/volumes"
Oct 30 20:46:58.768249 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:46:58.768216    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:00.801380 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:00.801342    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"]
Oct 30 20:47:00.802184 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:00.802166    5071 cpu_manager.go:414] "containerMap: removing container" podUID="5de98b75-5345-48cd-8b72-31d9f5a149b8" containerName="revoke-keys"
Oct 30 20:47:00.802227 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:00.802186    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5de98b75-5345-48cd-8b72-31d9f5a149b8" containerName="revoke-keys"
Oct 30 20:47:00.802311 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:00.802303    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="5de98b75-5345-48cd-8b72-31d9f5a149b8" containerName="revoke-keys"
Oct 30 20:47:00.806326 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:00.806307    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"
Oct 30 20:47:00.878795 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:00.878758    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8d10219-d907-47b1-be66-70a4e9d40465-workload-certs\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5\" (UID: \"b8d10219-d907-47b1-be66-70a4e9d40465\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"
Oct 30 20:47:00.878977 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:00.878835    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d10219-d907-47b1-be66-70a4e9d40465-workload-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5\" (UID: \"b8d10219-d907-47b1-be66-70a4e9d40465\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"
Oct 30 20:47:00.878977 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:00.878915    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8d10219-d907-47b1-be66-70a4e9d40465-istiod-ca-cert\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5\" (UID: \"b8d10219-d907-47b1-be66-70a4e9d40465\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"
Oct 30 20:47:00.878977 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:00.878946    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8d10219-d907-47b1-be66-70a4e9d40465-istio-envoy\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5\" (UID: \"b8d10219-d907-47b1-be66-70a4e9d40465\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"
Oct 30 20:47:00.878977 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:00.878966    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8d10219-d907-47b1-be66-70a4e9d40465-istio-data\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5\" (UID: \"b8d10219-d907-47b1-be66-70a4e9d40465\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"
Oct 30 20:47:00.879166 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:00.879143    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8d10219-d907-47b1-be66-70a4e9d40465-istio-token\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5\" (UID: \"b8d10219-d907-47b1-be66-70a4e9d40465\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"
Oct 30 20:47:00.879270 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:00.879255    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rfjt\" (UniqueName: \"kubernetes.io/projected/b8d10219-d907-47b1-be66-70a4e9d40465-kube-api-access-7rfjt\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5\" (UID: \"b8d10219-d907-47b1-be66-70a4e9d40465\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"
Oct 30 20:47:00.879318 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:00.879284    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8d10219-d907-47b1-be66-70a4e9d40465-istio-podinfo\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5\" (UID: \"b8d10219-d907-47b1-be66-70a4e9d40465\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"
Oct 30 20:47:00.879358 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:00.879339    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d10219-d907-47b1-be66-70a4e9d40465-credential-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5\" (UID: \"b8d10219-d907-47b1-be66-70a4e9d40465\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"
Oct 30 20:47:00.980279 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:00.980243    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8d10219-d907-47b1-be66-70a4e9d40465-istio-token\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5\" (UID: \"b8d10219-d907-47b1-be66-70a4e9d40465\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"
Oct 30 20:47:00.980485 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:00.980333    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rfjt\" (UniqueName: \"kubernetes.io/projected/b8d10219-d907-47b1-be66-70a4e9d40465-kube-api-access-7rfjt\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5\" (UID: \"b8d10219-d907-47b1-be66-70a4e9d40465\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"
Oct 30 20:47:00.980485 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:00.980362    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8d10219-d907-47b1-be66-70a4e9d40465-istio-podinfo\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5\" (UID: \"b8d10219-d907-47b1-be66-70a4e9d40465\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"
Oct 30 20:47:00.980485 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:00.980382    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d10219-d907-47b1-be66-70a4e9d40465-credential-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5\" (UID: \"b8d10219-d907-47b1-be66-70a4e9d40465\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"
Oct 30 20:47:00.980485 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:00.980424    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8d10219-d907-47b1-be66-70a4e9d40465-workload-certs\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5\" (UID: \"b8d10219-d907-47b1-be66-70a4e9d40465\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"
Oct 30 20:47:00.980738 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:00.980489    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d10219-d907-47b1-be66-70a4e9d40465-workload-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5\" (UID: \"b8d10219-d907-47b1-be66-70a4e9d40465\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"
Oct 30 20:47:00.980738 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:00.980526    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8d10219-d907-47b1-be66-70a4e9d40465-istiod-ca-cert\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5\" (UID: \"b8d10219-d907-47b1-be66-70a4e9d40465\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"
Oct 30 20:47:00.980738 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:00.980558    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8d10219-d907-47b1-be66-70a4e9d40465-istio-envoy\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5\" (UID: \"b8d10219-d907-47b1-be66-70a4e9d40465\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"
Oct 30 20:47:00.980738 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:00.980582    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8d10219-d907-47b1-be66-70a4e9d40465-istio-data\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5\" (UID: \"b8d10219-d907-47b1-be66-70a4e9d40465\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"
Oct 30 20:47:00.981008 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:00.980987    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8d10219-d907-47b1-be66-70a4e9d40465-workload-certs\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5\" (UID: \"b8d10219-d907-47b1-be66-70a4e9d40465\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"
Oct 30 20:47:00.981074 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:00.981026    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8d10219-d907-47b1-be66-70a4e9d40465-istio-data\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5\" (UID: \"b8d10219-d907-47b1-be66-70a4e9d40465\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"
Oct 30 20:47:00.981475 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:00.981449    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d10219-d907-47b1-be66-70a4e9d40465-credential-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5\" (UID: \"b8d10219-d907-47b1-be66-70a4e9d40465\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"
Oct 30 20:47:00.981641 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:00.981451    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8d10219-d907-47b1-be66-70a4e9d40465-istiod-ca-cert\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5\" (UID: \"b8d10219-d907-47b1-be66-70a4e9d40465\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"
Oct 30 20:47:00.981641 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:00.981542    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d10219-d907-47b1-be66-70a4e9d40465-workload-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5\" (UID: \"b8d10219-d907-47b1-be66-70a4e9d40465\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"
Oct 30 20:47:00.984919 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:00.984894    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8d10219-d907-47b1-be66-70a4e9d40465-istio-podinfo\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5\" (UID: \"b8d10219-d907-47b1-be66-70a4e9d40465\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"
Oct 30 20:47:00.985131 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:00.985105    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8d10219-d907-47b1-be66-70a4e9d40465-istio-envoy\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5\" (UID: \"b8d10219-d907-47b1-be66-70a4e9d40465\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"
Oct 30 20:47:01.016678 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:01.016645    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"]
Oct 30 20:47:01.039308 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:01.039278    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8d10219-d907-47b1-be66-70a4e9d40465-istio-token\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5\" (UID: \"b8d10219-d907-47b1-be66-70a4e9d40465\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"
Oct 30 20:47:01.039611 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:01.039579    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rfjt\" (UniqueName: \"kubernetes.io/projected/b8d10219-d907-47b1-be66-70a4e9d40465-kube-api-access-7rfjt\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5\" (UID: \"b8d10219-d907-47b1-be66-70a4e9d40465\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"
Oct 30 20:47:01.119988 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:01.119901    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"
Oct 30 20:47:01.560375 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:01.560344    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"]
Oct 30 20:47:01.561292 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:01.561256    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:47:01.561424 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:01.561325    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:47:01.561424 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:01.561355    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:47:02.024093 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:02.024003    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5" event={"ID":"b8d10219-d907-47b1-be66-70a4e9d40465","Type":"ContainerStarted","Data":"a900003585f1a457a8d53a69adc4c1f492f4d44f31ebb02cefb7fff6ac2c2183"}
Oct 30 20:47:02.024093 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:02.024053    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5" event={"ID":"b8d10219-d907-47b1-be66-70a4e9d40465","Type":"ContainerStarted","Data":"b97a0a015edd4d1c30848d608bd4fc158626e390503072444557c99ed7dd66a2"}
Oct 30 20:47:02.054543 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:02.054511    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3041002-jsp28"]
Oct 30 20:47:02.102944 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:02.102913    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3041002-jsp28"]
Oct 30 20:47:02.120390 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:02.120363    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"
Oct 30 20:47:02.121899 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:02.121871    5071 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 20:47:02.122018 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:02.121912    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5" podUID="b8d10219-d907-47b1-be66-70a4e9d40465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 20:47:02.168381 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:02.168320    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5" podStartSLOduration=2.168305524 podStartE2EDuration="2.168305524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:02.164624592 +0000 UTC m=+1234.081646128" watchObservedRunningTime="2026-10-30 20:47:02.168305524 +0000 UTC m=+1234.085327059"
Oct 30 20:47:02.203996 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:02.203961    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36b847ec-944b-442b-a43d-7334f32eba81" path="/var/lib/kubelet/pods/36b847ec-944b-442b-a43d-7334f32eba81/volumes"
Oct 30 20:47:03.120360 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:03.120326    5071 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 20:47:03.120844 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:03.120381    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5" podUID="b8d10219-d907-47b1-be66-70a4e9d40465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 20:47:03.791208 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:03.791174    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:04.121927 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:04.121456    5071 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 20:47:04.121927 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:04.121527    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5" podUID="b8d10219-d907-47b1-be66-70a4e9d40465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 20:47:05.160848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:05.143258    5071 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 20:47:05.160848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:05.143326    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5" podUID="b8d10219-d907-47b1-be66-70a4e9d40465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 20:47:05.435036 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:05.434924    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:05.779910 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:05.776895    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:06.133696 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:06.133213    5071 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 20:47:06.133696 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:06.133293    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5" podUID="b8d10219-d907-47b1-be66-70a4e9d40465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 20:47:06.317848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:06.304962    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:06.437055 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:06.436951    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:06.438000 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:06.437975    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:47:06.784907 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:06.781960    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:06.784907 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:06.782736    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:47:06.859864 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:06.810886    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 20:47:06.859864 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:06.847202    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:07.422706 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:07.309092    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:07.422706 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:07.309521    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:47:07.458733 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:07.437955    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:07.594675 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:07.594634    5071 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 20:47:07.594968 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:07.594934    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5" podUID="b8d10219-d907-47b1-be66-70a4e9d40465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 20:47:07.756025 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:07.755928    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:47:07.764856 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:07.756264    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87" podUID="723f1905-6d5a-45a7-a2bb-d5753a0fda9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:07.842046 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:07.788748    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:07.862321 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:07.816117    5071 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:07.862321 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:07.816165    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:07.862321 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:07.848419    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:07.873853 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:07.863087    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:47:07.873853 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:07.863149    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:47:08.164902 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:08.134247    5071 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 20:47:08.164902 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:08.134302    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5" podUID="b8d10219-d907-47b1-be66-70a4e9d40465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 20:47:08.322852 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:08.320608    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 30 20:47:08.465516 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:08.465416    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:08.893875 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:08.875919    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:08.893875 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:08.876293    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:08.893875 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:08.876632    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:09.091978 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:09.091934    5071 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:09.133170 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:09.133135    5071 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 20:47:09.133463 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:09.133425    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5" podUID="b8d10219-d907-47b1-be66-70a4e9d40465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 20:47:09.306574 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:09.306543    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:47:09.337164 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:09.337136    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:47:09.428069 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:09.428038    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:47:09.455424 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:09.455387    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:47:09.504958 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:09.504929    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:47:10.121393 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:10.121362    5071 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 20:47:10.121572 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:10.121432    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5" podUID="b8d10219-d907-47b1-be66-70a4e9d40465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 20:47:10.625201 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:10.625163    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:11.120568 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:11.120535    5071 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 20:47:11.120738 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:11.120591    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5" podUID="b8d10219-d907-47b1-be66-70a4e9d40465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 20:47:12.327933 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:12.327876    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"
Oct 30 20:47:12.329474 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:12.329451    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"
Oct 30 20:47:12.330413 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:12.330380    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"
Oct 30 20:47:12.673551 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:12.673451    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d5a3a4c-8r7gr"]
Oct 30 20:47:12.679837 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:12.679799    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5a3a4c-8r7gr"
Oct 30 20:47:12.712914 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:12.712884    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d5a3a4c-8r7gr"]
Oct 30 20:47:12.733859 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:12.733796    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:47:12.734014 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:12.733877    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 20:47:12.783218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:12.783176    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10b290b3-522a-4300-a597-1835714ebda5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d5a3a4c-8r7gr\" (UID: \"10b290b3-522a-4300-a597-1835714ebda5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5a3a4c-8r7gr"
Oct 30 20:47:12.783512 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:12.783493    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10b290b3-522a-4300-a597-1835714ebda5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d5a3a4c-8r7gr\" (UID: \"10b290b3-522a-4300-a597-1835714ebda5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5a3a4c-8r7gr"
Oct 30 20:47:12.783639 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:12.783624    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fhj2\" (UniqueName: \"kubernetes.io/projected/10b290b3-522a-4300-a597-1835714ebda5-kube-api-access-4fhj2\") pod \"maas-api-revoke-sub-2d5a3a4c-8r7gr\" (UID: \"10b290b3-522a-4300-a597-1835714ebda5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5a3a4c-8r7gr"
Oct 30 20:47:12.884969 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:12.884932    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10b290b3-522a-4300-a597-1835714ebda5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d5a3a4c-8r7gr\" (UID: \"10b290b3-522a-4300-a597-1835714ebda5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5a3a4c-8r7gr"
Oct 30 20:47:12.885160 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:12.884981    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fhj2\" (UniqueName: \"kubernetes.io/projected/10b290b3-522a-4300-a597-1835714ebda5-kube-api-access-4fhj2\") pod \"maas-api-revoke-sub-2d5a3a4c-8r7gr\" (UID: \"10b290b3-522a-4300-a597-1835714ebda5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5a3a4c-8r7gr"
Oct 30 20:47:12.885265 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:12.885249    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10b290b3-522a-4300-a597-1835714ebda5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d5a3a4c-8r7gr\" (UID: \"10b290b3-522a-4300-a597-1835714ebda5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5a3a4c-8r7gr"
Oct 30 20:47:12.886066 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:12.886042    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10b290b3-522a-4300-a597-1835714ebda5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d5a3a4c-8r7gr\" (UID: \"10b290b3-522a-4300-a597-1835714ebda5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5a3a4c-8r7gr"
Oct 30 20:47:12.887622 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:12.887599    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10b290b3-522a-4300-a597-1835714ebda5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d5a3a4c-8r7gr\" (UID: \"10b290b3-522a-4300-a597-1835714ebda5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5a3a4c-8r7gr"
Oct 30 20:47:12.945764 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:12.945682    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fhj2\" (UniqueName: \"kubernetes.io/projected/10b290b3-522a-4300-a597-1835714ebda5-kube-api-access-4fhj2\") pod \"maas-api-revoke-sub-2d5a3a4c-8r7gr\" (UID: \"10b290b3-522a-4300-a597-1835714ebda5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5a3a4c-8r7gr"
Oct 30 20:47:12.983359 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:12.983324    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25bb2df2-grdd7"]
Oct 30 20:47:12.988059 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:12.988038    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25bb2df2-grdd7"
Oct 30 20:47:12.995445 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:12.995421    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5a3a4c-8r7gr"
Oct 30 20:47:13.052192 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:13.052105    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25bb2df2-grdd7"]
Oct 30 20:47:13.063901 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:13.063336    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1313706-889b7"]
Oct 30 20:47:13.069920 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:13.069892    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6a5c3ab-k6cfs"]
Oct 30 20:47:13.076567 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:13.076540    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1313706-889b7"]
Oct 30 20:47:13.084454 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:13.083163    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6a5c3ab-k6cfs"]
Oct 30 20:47:13.087466 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:13.087437    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25bb2df2-grdd7\" (UID: \"f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25bb2df2-grdd7"
Oct 30 20:47:13.087569 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:13.087480    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25bb2df2-grdd7\" (UID: \"f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25bb2df2-grdd7"
Oct 30 20:47:13.087636 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:13.087619    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tkwg\" (UniqueName: \"kubernetes.io/projected/f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea-kube-api-access-5tkwg\") pod \"maas-api-revoke-sub-25bb2df2-grdd7\" (UID: \"f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25bb2df2-grdd7"
Oct 30 20:47:13.188956 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:13.188902    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25bb2df2-grdd7\" (UID: \"f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25bb2df2-grdd7"
Oct 30 20:47:13.188956 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:13.188958    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25bb2df2-grdd7\" (UID: \"f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25bb2df2-grdd7"
Oct 30 20:47:13.189166 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:13.189048    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tkwg\" (UniqueName: \"kubernetes.io/projected/f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea-kube-api-access-5tkwg\") pod \"maas-api-revoke-sub-25bb2df2-grdd7\" (UID: \"f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25bb2df2-grdd7"
Oct 30 20:47:13.189684 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:13.189659    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25bb2df2-grdd7\" (UID: \"f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25bb2df2-grdd7"
Oct 30 20:47:13.191495 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:13.191477    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25bb2df2-grdd7\" (UID: \"f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25bb2df2-grdd7"
Oct 30 20:47:13.248975 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:13.248899    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tkwg\" (UniqueName: \"kubernetes.io/projected/f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea-kube-api-access-5tkwg\") pod \"maas-api-revoke-sub-25bb2df2-grdd7\" (UID: \"f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25bb2df2-grdd7"
Oct 30 20:47:13.303140 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:13.303110    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25bb2df2-grdd7"
Oct 30 20:47:13.401333 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:47:13.401300    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b290b3_522a_4300_a597_1835714ebda5.slice/crio-e8e5906abb1b83b03615754f99a3cbb211206bdf76ecd0897171a2fc882be4eb WatchSource:0}: Error finding container e8e5906abb1b83b03615754f99a3cbb211206bdf76ecd0897171a2fc882be4eb: Status 404 returned error can't find the container with id e8e5906abb1b83b03615754f99a3cbb211206bdf76ecd0897171a2fc882be4eb
Oct 30 20:47:13.404384 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:13.404364    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d5a3a4c-8r7gr"]
Oct 30 20:47:13.688011 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:13.687985    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25bb2df2-grdd7"]
Oct 30 20:47:13.688478 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:47:13.688454    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46b43fd_d482_4a4e_a4fe_dce7f9a7e2ea.slice/crio-1dcea24518046ecc33d55fca9f956908304e94f4efff0b7111829184e26e04f9 WatchSource:0}: Error finding container 1dcea24518046ecc33d55fca9f956908304e94f4efff0b7111829184e26e04f9: Status 404 returned error can't find the container with id 1dcea24518046ecc33d55fca9f956908304e94f4efff0b7111829184e26e04f9
Oct 30 20:47:13.917443 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:13.917350    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25bb2df2-grdd7" event={"ID":"f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea","Type":"ContainerStarted","Data":"1dcea24518046ecc33d55fca9f956908304e94f4efff0b7111829184e26e04f9"}
Oct 30 20:47:13.919788 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:13.919750    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5a3a4c-8r7gr" event={"ID":"10b290b3-522a-4300-a597-1835714ebda5","Type":"ContainerStarted","Data":"f7319e70e47d2bb6c1d7e4d5041f3aeb1a6de503d637a057665a66acb7dc687e"}
Oct 30 20:47:13.919946 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:13.919812    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5a3a4c-8r7gr" event={"ID":"10b290b3-522a-4300-a597-1835714ebda5","Type":"ContainerStarted","Data":"e8e5906abb1b83b03615754f99a3cbb211206bdf76ecd0897171a2fc882be4eb"}
Oct 30 20:47:14.040651 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:14.040599    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5a3a4c-8r7gr" podStartSLOduration=2.040586203 podStartE2EDuration="2.040586203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:14.035595809 +0000 UTC m=+1245.952617347" watchObservedRunningTime="2026-10-30 20:47:14.040586203 +0000 UTC m=+1245.957607739"
Oct 30 20:47:14.202436 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:14.202353    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01b55b11-863d-4088-a049-29480079f206" path="/var/lib/kubelet/pods/01b55b11-863d-4088-a049-29480079f206/volumes"
Oct 30 20:47:14.202833 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:14.202804    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c977357-4ddf-4cfc-925b-7019c62edee8" path="/var/lib/kubelet/pods/2c977357-4ddf-4cfc-925b-7019c62edee8/volumes"
Oct 30 20:47:14.928896 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:14.928776    5071 generic.go:356] "Generic (PLEG): container finished" podID="f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea" containerID="9dd405ea783d5a5ccf6243ab86b5f44f88e2da9d369b0352984fd04c2cfdcf06" exitCode=0
Oct 30 20:47:14.928896 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:14.928860    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25bb2df2-grdd7" event={"ID":"f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea","Type":"ContainerDied","Data":"9dd405ea783d5a5ccf6243ab86b5f44f88e2da9d369b0352984fd04c2cfdcf06"}
Oct 30 20:47:14.930908 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:14.930880    5071 generic.go:356] "Generic (PLEG): container finished" podID="10b290b3-522a-4300-a597-1835714ebda5" containerID="f7319e70e47d2bb6c1d7e4d5041f3aeb1a6de503d637a057665a66acb7dc687e" exitCode=0
Oct 30 20:47:14.931027 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:14.930908    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5a3a4c-8r7gr" event={"ID":"10b290b3-522a-4300-a597-1835714ebda5","Type":"ContainerDied","Data":"f7319e70e47d2bb6c1d7e4d5041f3aeb1a6de503d637a057665a66acb7dc687e"}
Oct 30 20:47:15.683624 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:15.683590    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:16.143118 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:16.143085    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25bb2df2-grdd7"
Oct 30 20:47:16.206207 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:16.206139    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5a3a4c-8r7gr"
Oct 30 20:47:16.238528 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:16.238503    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10b290b3-522a-4300-a597-1835714ebda5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10b290b3-522a-4300-a597-1835714ebda5-maas-api-xks-ca\") pod \"10b290b3-522a-4300-a597-1835714ebda5\" (UID: \"10b290b3-522a-4300-a597-1835714ebda5\") "
Oct 30 20:47:16.238695 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:16.238619    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea-maas-api-xks-ca\") pod \"f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea\" (UID: \"f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea\") "
Oct 30 20:47:16.238765 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:16.238701    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea-maas-api-openshift-ca\") pod \"f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea\" (UID: \"f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea\") "
Oct 30 20:47:16.238765 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:16.238741    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10b290b3-522a-4300-a597-1835714ebda5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10b290b3-522a-4300-a597-1835714ebda5-maas-api-openshift-ca\") pod \"10b290b3-522a-4300-a597-1835714ebda5\" (UID: \"10b290b3-522a-4300-a597-1835714ebda5\") "
Oct 30 20:47:16.238929 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:16.238814    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b290b3-522a-4300-a597-1835714ebda5-kube-api-access-4fhj2\" (UniqueName: \"kubernetes.io/projected/10b290b3-522a-4300-a597-1835714ebda5-kube-api-access-4fhj2\") pod \"10b290b3-522a-4300-a597-1835714ebda5\" (UID: \"10b290b3-522a-4300-a597-1835714ebda5\") "
Oct 30 20:47:16.238929 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:16.238880    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea-kube-api-access-5tkwg\" (UniqueName: \"kubernetes.io/projected/f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea-kube-api-access-5tkwg\") pod \"f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea\" (UID: \"f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea\") "
Oct 30 20:47:16.241628 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:16.239433    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10b290b3-522a-4300-a597-1835714ebda5-maas-api-openshift-ca" pod "10b290b3-522a-4300-a597-1835714ebda5" (UID: "10b290b3-522a-4300-a597-1835714ebda5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:47:16.241628 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:16.239870    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea-maas-api-openshift-ca" pod "f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea" (UID: "f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:47:16.243011 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:16.242948    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:16.243011 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:16.242978    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10b290b3-522a-4300-a597-1835714ebda5-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:16.255911 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:16.255629    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea-kube-api-access-5tkwg" pod "f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea" (UID: "f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea"). InnerVolumeSpecName "kube-api-access-5tkwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:16.256007 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:16.255929    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea-maas-api-xks-ca" pod "f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea" (UID: "f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:47:16.256078 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:16.256009    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10b290b3-522a-4300-a597-1835714ebda5-maas-api-xks-ca" pod "10b290b3-522a-4300-a597-1835714ebda5" (UID: "10b290b3-522a-4300-a597-1835714ebda5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:47:16.257495 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:16.257471    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b290b3-522a-4300-a597-1835714ebda5-kube-api-access-4fhj2" pod "10b290b3-522a-4300-a597-1835714ebda5" (UID: "10b290b3-522a-4300-a597-1835714ebda5"). InnerVolumeSpecName "kube-api-access-4fhj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:16.344644 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:16.344606    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fhj2\" (UniqueName: \"kubernetes.io/projected/10b290b3-522a-4300-a597-1835714ebda5-kube-api-access-4fhj2\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:16.344644 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:16.344639    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tkwg\" (UniqueName: \"kubernetes.io/projected/f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea-kube-api-access-5tkwg\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:16.344644 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:16.344650    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10b290b3-522a-4300-a597-1835714ebda5-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:16.344914 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:16.344660    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:16.729633 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:47:16.729380    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:47:16.950092 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:16.950062    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5a3a4c-8r7gr"
Oct 30 20:47:16.950288 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:16.950098    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5a3a4c-8r7gr" event={"ID":"10b290b3-522a-4300-a597-1835714ebda5","Type":"ContainerDied","Data":"e8e5906abb1b83b03615754f99a3cbb211206bdf76ecd0897171a2fc882be4eb"}
Oct 30 20:47:16.950288 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:16.950165    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8e5906abb1b83b03615754f99a3cbb211206bdf76ecd0897171a2fc882be4eb"
Oct 30 20:47:16.951656 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:16.951632    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25bb2df2-grdd7" event={"ID":"f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea","Type":"ContainerDied","Data":"1dcea24518046ecc33d55fca9f956908304e94f4efff0b7111829184e26e04f9"}
Oct 30 20:47:16.951797 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:16.951667    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1dcea24518046ecc33d55fca9f956908304e94f4efff0b7111829184e26e04f9"
Oct 30 20:47:16.951797 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:16.951681    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25bb2df2-grdd7"
Oct 30 20:47:18.055795 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:18.055765    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f4d9d61-5bqhp"]
Oct 30 20:47:18.104794 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:18.104763    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f4d9d61-5bqhp"]
Oct 30 20:47:18.203893 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:18.203864    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c1c144f-aa0b-4d88-a3a5-4c399288fd1c" path="/var/lib/kubelet/pods/5c1c144f-aa0b-4d88-a3a5-4c399288fd1c/volumes"
Oct 30 20:47:18.484517 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:18.484438    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af25ff0e-2kjp7"]
Oct 30 20:47:18.485085 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:18.485071    5071 cpu_manager.go:414] "containerMap: removing container" podUID="10b290b3-522a-4300-a597-1835714ebda5" containerName="revoke-keys"
Oct 30 20:47:18.485127 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:18.485087    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10b290b3-522a-4300-a597-1835714ebda5" containerName="revoke-keys"
Oct 30 20:47:18.485127 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:18.485113    5071 cpu_manager.go:414] "containerMap: removing container" podUID="f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea" containerName="revoke-keys"
Oct 30 20:47:18.485127 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:18.485117    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea" containerName="revoke-keys"
Oct 30 20:47:18.485234 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:18.485225    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="10b290b3-522a-4300-a597-1835714ebda5" containerName="revoke-keys"
Oct 30 20:47:18.485270 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:18.485239    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea" containerName="revoke-keys"
Oct 30 20:47:18.489999 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:18.489981    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af25ff0e-2kjp7"
Oct 30 20:47:18.553903 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:18.553869    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af25ff0e-2kjp7"]
Oct 30 20:47:18.571983 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:18.571948    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d60d9465-5025-4279-806f-f7a19e55630a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af25ff0e-2kjp7\" (UID: \"d60d9465-5025-4279-806f-f7a19e55630a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af25ff0e-2kjp7"
Oct 30 20:47:18.572178 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:18.572123    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d60d9465-5025-4279-806f-f7a19e55630a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af25ff0e-2kjp7\" (UID: \"d60d9465-5025-4279-806f-f7a19e55630a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af25ff0e-2kjp7"
Oct 30 20:47:18.572307 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:18.572284    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsdwm\" (UniqueName: \"kubernetes.io/projected/d60d9465-5025-4279-806f-f7a19e55630a-kube-api-access-xsdwm\") pod \"maas-api-revoke-sub-af25ff0e-2kjp7\" (UID: \"d60d9465-5025-4279-806f-f7a19e55630a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af25ff0e-2kjp7"
Oct 30 20:47:18.597928 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:18.597898    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:47:18.598079 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:18.597960    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 20:47:18.673851 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:18.673805    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsdwm\" (UniqueName: \"kubernetes.io/projected/d60d9465-5025-4279-806f-f7a19e55630a-kube-api-access-xsdwm\") pod \"maas-api-revoke-sub-af25ff0e-2kjp7\" (UID: \"d60d9465-5025-4279-806f-f7a19e55630a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af25ff0e-2kjp7"
Oct 30 20:47:18.674031 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:18.673975    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d60d9465-5025-4279-806f-f7a19e55630a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af25ff0e-2kjp7\" (UID: \"d60d9465-5025-4279-806f-f7a19e55630a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af25ff0e-2kjp7"
Oct 30 20:47:18.674079 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:18.674051    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d60d9465-5025-4279-806f-f7a19e55630a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af25ff0e-2kjp7\" (UID: \"d60d9465-5025-4279-806f-f7a19e55630a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af25ff0e-2kjp7"
Oct 30 20:47:18.674716 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:18.674699    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d60d9465-5025-4279-806f-f7a19e55630a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af25ff0e-2kjp7\" (UID: \"d60d9465-5025-4279-806f-f7a19e55630a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af25ff0e-2kjp7"
Oct 30 20:47:18.676712 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:18.676678    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d60d9465-5025-4279-806f-f7a19e55630a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af25ff0e-2kjp7\" (UID: \"d60d9465-5025-4279-806f-f7a19e55630a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af25ff0e-2kjp7"
Oct 30 20:47:18.732959 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:18.732927    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsdwm\" (UniqueName: \"kubernetes.io/projected/d60d9465-5025-4279-806f-f7a19e55630a-kube-api-access-xsdwm\") pod \"maas-api-revoke-sub-af25ff0e-2kjp7\" (UID: \"d60d9465-5025-4279-806f-f7a19e55630a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af25ff0e-2kjp7"
Oct 30 20:47:18.803077 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:18.803046    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af25ff0e-2kjp7"
Oct 30 20:47:19.195459 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:47:19.195427    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd60d9465_5025_4279_806f_f7a19e55630a.slice/crio-031218846c1d4c3ef47bd15058e867c3f75ee7fa09fc1aa48f62eae7aeeae80a WatchSource:0}: Error finding container 031218846c1d4c3ef47bd15058e867c3f75ee7fa09fc1aa48f62eae7aeeae80a: Status 404 returned error can't find the container with id 031218846c1d4c3ef47bd15058e867c3f75ee7fa09fc1aa48f62eae7aeeae80a
Oct 30 20:47:19.195809 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:19.195603    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af25ff0e-2kjp7"]
Oct 30 20:47:19.989912 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:19.989866    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af25ff0e-2kjp7" event={"ID":"d60d9465-5025-4279-806f-f7a19e55630a","Type":"ContainerStarted","Data":"05249dde4b9db70a45776c3fc32d921cea79b82af00228c84fef0a686152725d"}
Oct 30 20:47:19.990122 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:19.989965    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af25ff0e-2kjp7" event={"ID":"d60d9465-5025-4279-806f-f7a19e55630a","Type":"ContainerStarted","Data":"031218846c1d4c3ef47bd15058e867c3f75ee7fa09fc1aa48f62eae7aeeae80a"}
Oct 30 20:47:20.108402 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:20.108352    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af25ff0e-2kjp7" podStartSLOduration=2.108338198 podStartE2EDuration="2.108338198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:20.105669135 +0000 UTC m=+1252.022690671" watchObservedRunningTime="2026-10-30 20:47:20.108338198 +0000 UTC m=+1252.025359734"
Oct 30 20:47:20.707847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:20.683332    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:20.999887 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:20.999798    5071 generic.go:356] "Generic (PLEG): container finished" podID="d60d9465-5025-4279-806f-f7a19e55630a" containerID="05249dde4b9db70a45776c3fc32d921cea79b82af00228c84fef0a686152725d" exitCode=0
Oct 30 20:47:20.999887 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:20.999872    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af25ff0e-2kjp7" event={"ID":"d60d9465-5025-4279-806f-f7a19e55630a","Type":"ContainerDied","Data":"05249dde4b9db70a45776c3fc32d921cea79b82af00228c84fef0a686152725d"}
Oct 30 20:47:22.198062 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:22.198037    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af25ff0e-2kjp7"
Oct 30 20:47:22.321900 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:22.321866    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d60d9465-5025-4279-806f-f7a19e55630a-kube-api-access-xsdwm\" (UniqueName: \"kubernetes.io/projected/d60d9465-5025-4279-806f-f7a19e55630a-kube-api-access-xsdwm\") pod \"d60d9465-5025-4279-806f-f7a19e55630a\" (UID: \"d60d9465-5025-4279-806f-f7a19e55630a\") "
Oct 30 20:47:22.322096 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:22.321955    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d60d9465-5025-4279-806f-f7a19e55630a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d60d9465-5025-4279-806f-f7a19e55630a-maas-api-openshift-ca\") pod \"d60d9465-5025-4279-806f-f7a19e55630a\" (UID: \"d60d9465-5025-4279-806f-f7a19e55630a\") "
Oct 30 20:47:22.322096 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:22.322040    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d60d9465-5025-4279-806f-f7a19e55630a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d60d9465-5025-4279-806f-f7a19e55630a-maas-api-xks-ca\") pod \"d60d9465-5025-4279-806f-f7a19e55630a\" (UID: \"d60d9465-5025-4279-806f-f7a19e55630a\") "
Oct 30 20:47:22.324268 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:22.322299    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d60d9465-5025-4279-806f-f7a19e55630a-maas-api-openshift-ca" pod "d60d9465-5025-4279-806f-f7a19e55630a" (UID: "d60d9465-5025-4279-806f-f7a19e55630a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:47:22.324268 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:22.322806    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d60d9465-5025-4279-806f-f7a19e55630a-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:22.324721 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:22.324683    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d60d9465-5025-4279-806f-f7a19e55630a-kube-api-access-xsdwm" pod "d60d9465-5025-4279-806f-f7a19e55630a" (UID: "d60d9465-5025-4279-806f-f7a19e55630a"). InnerVolumeSpecName "kube-api-access-xsdwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:22.324721 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:22.324681    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d60d9465-5025-4279-806f-f7a19e55630a-maas-api-xks-ca" pod "d60d9465-5025-4279-806f-f7a19e55630a" (UID: "d60d9465-5025-4279-806f-f7a19e55630a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:47:22.424363 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:22.424328    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsdwm\" (UniqueName: \"kubernetes.io/projected/d60d9465-5025-4279-806f-f7a19e55630a-kube-api-access-xsdwm\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:22.424363 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:22.424362    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d60d9465-5025-4279-806f-f7a19e55630a-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:23.018468 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:23.018428    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af25ff0e-2kjp7"
Oct 30 20:47:23.018683 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:23.018467    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af25ff0e-2kjp7" event={"ID":"d60d9465-5025-4279-806f-f7a19e55630a","Type":"ContainerDied","Data":"031218846c1d4c3ef47bd15058e867c3f75ee7fa09fc1aa48f62eae7aeeae80a"}
Oct 30 20:47:23.018683 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:23.018527    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="031218846c1d4c3ef47bd15058e867c3f75ee7fa09fc1aa48f62eae7aeeae80a"
Oct 30 20:47:25.765080 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:25.765042    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:25.894090 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:25.894060    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-066357ca-mb8dn"]
Oct 30 20:47:25.894681 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:25.894667    5071 cpu_manager.go:414] "containerMap: removing container" podUID="d60d9465-5025-4279-806f-f7a19e55630a" containerName="revoke-keys"
Oct 30 20:47:25.894738 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:25.894682    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d60d9465-5025-4279-806f-f7a19e55630a" containerName="revoke-keys"
Oct 30 20:47:25.894845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:25.894816    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="d60d9465-5025-4279-806f-f7a19e55630a" containerName="revoke-keys"
Oct 30 20:47:25.898757 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:25.898740    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-066357ca-mb8dn"
Oct 30 20:47:25.956135 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:25.956096    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-066357ca-mb8dn"]
Oct 30 20:47:26.000493 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:26.000464    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:47:26.000687 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:26.000570    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 20:47:26.074238 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:26.074208    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90acbd64-4882-4fce-92cc-7e8b1e4ad884-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-066357ca-mb8dn\" (UID: \"90acbd64-4882-4fce-92cc-7e8b1e4ad884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-066357ca-mb8dn"
Oct 30 20:47:26.074409 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:26.074247    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90acbd64-4882-4fce-92cc-7e8b1e4ad884-maas-api-xks-ca\") pod \"maas-api-revoke-sub-066357ca-mb8dn\" (UID: \"90acbd64-4882-4fce-92cc-7e8b1e4ad884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-066357ca-mb8dn"
Oct 30 20:47:26.074451 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:26.074444    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-296ck\" (UniqueName: \"kubernetes.io/projected/90acbd64-4882-4fce-92cc-7e8b1e4ad884-kube-api-access-296ck\") pod \"maas-api-revoke-sub-066357ca-mb8dn\" (UID: \"90acbd64-4882-4fce-92cc-7e8b1e4ad884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-066357ca-mb8dn"
Oct 30 20:47:26.175891 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:26.175848    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-296ck\" (UniqueName: \"kubernetes.io/projected/90acbd64-4882-4fce-92cc-7e8b1e4ad884-kube-api-access-296ck\") pod \"maas-api-revoke-sub-066357ca-mb8dn\" (UID: \"90acbd64-4882-4fce-92cc-7e8b1e4ad884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-066357ca-mb8dn"
Oct 30 20:47:26.176066 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:26.176003    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90acbd64-4882-4fce-92cc-7e8b1e4ad884-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-066357ca-mb8dn\" (UID: \"90acbd64-4882-4fce-92cc-7e8b1e4ad884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-066357ca-mb8dn"
Oct 30 20:47:26.176066 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:26.176041    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90acbd64-4882-4fce-92cc-7e8b1e4ad884-maas-api-xks-ca\") pod \"maas-api-revoke-sub-066357ca-mb8dn\" (UID: \"90acbd64-4882-4fce-92cc-7e8b1e4ad884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-066357ca-mb8dn"
Oct 30 20:47:26.176668 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:26.176646    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90acbd64-4882-4fce-92cc-7e8b1e4ad884-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-066357ca-mb8dn\" (UID: \"90acbd64-4882-4fce-92cc-7e8b1e4ad884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-066357ca-mb8dn"
Oct 30 20:47:26.178676 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:26.178641    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90acbd64-4882-4fce-92cc-7e8b1e4ad884-maas-api-xks-ca\") pod \"maas-api-revoke-sub-066357ca-mb8dn\" (UID: \"90acbd64-4882-4fce-92cc-7e8b1e4ad884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-066357ca-mb8dn"
Oct 30 20:47:26.234431 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:26.234396    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-296ck\" (UniqueName: \"kubernetes.io/projected/90acbd64-4882-4fce-92cc-7e8b1e4ad884-kube-api-access-296ck\") pod \"maas-api-revoke-sub-066357ca-mb8dn\" (UID: \"90acbd64-4882-4fce-92cc-7e8b1e4ad884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-066357ca-mb8dn"
Oct 30 20:47:26.513167 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:26.513095    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-066357ca-mb8dn"
Oct 30 20:47:26.896041 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:26.896008    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-066357ca-mb8dn"]
Oct 30 20:47:26.896479 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:47:26.896246    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90acbd64_4882_4fce_92cc_7e8b1e4ad884.slice/crio-65c3de861617a2a35ed956acad9710f67b31f2918e7c9c60e734c0c4e1bca174 WatchSource:0}: Error finding container 65c3de861617a2a35ed956acad9710f67b31f2918e7c9c60e734c0c4e1bca174: Status 404 returned error can't find the container with id 65c3de861617a2a35ed956acad9710f67b31f2918e7c9c60e734c0c4e1bca174
Oct 30 20:47:27.052614 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:27.052585    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-066357ca-mb8dn" event={"ID":"90acbd64-4882-4fce-92cc-7e8b1e4ad884","Type":"ContainerStarted","Data":"65c3de861617a2a35ed956acad9710f67b31f2918e7c9c60e734c0c4e1bca174"}
Oct 30 20:47:27.441578 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:27.441482    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:28.063111 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:28.063028    5071 generic.go:356] "Generic (PLEG): container finished" podID="90acbd64-4882-4fce-92cc-7e8b1e4ad884" containerID="42ce92727881463606893e301b3dccd7fb0b9102d0b52fd79d6f7d96a9d3d378" exitCode=0
Oct 30 20:47:28.063523 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:28.063122    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-066357ca-mb8dn" event={"ID":"90acbd64-4882-4fce-92cc-7e8b1e4ad884","Type":"ContainerDied","Data":"42ce92727881463606893e301b3dccd7fb0b9102d0b52fd79d6f7d96a9d3d378"}
Oct 30 20:47:28.406014 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:28.405909    5071 scope.go:122] "RemoveContainer" containerID="73c5064ecab9e10f7e206920e193e48a9f7ae3561df213cc555b6bb7894a3d45"
Oct 30 20:47:28.416590 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:28.416568    5071 scope.go:122] "RemoveContainer" containerID="b8f5007468b23439952abf0e3549d64ca1161531d491f77aeb0265a56b78b5b2"
Oct 30 20:47:28.427534 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:28.427507    5071 scope.go:122] "RemoveContainer" containerID="fded835e8bcc3b3685900599d605c285be4b29e5845c370cbecb316d6dea0d44"
Oct 30 20:47:29.268747 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:29.268723    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-066357ca-mb8dn"
Oct 30 20:47:29.316931 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:29.316899    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90acbd64-4882-4fce-92cc-7e8b1e4ad884-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90acbd64-4882-4fce-92cc-7e8b1e4ad884-maas-api-xks-ca\") pod \"90acbd64-4882-4fce-92cc-7e8b1e4ad884\" (UID: \"90acbd64-4882-4fce-92cc-7e8b1e4ad884\") "
Oct 30 20:47:29.317083 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:29.317005    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90acbd64-4882-4fce-92cc-7e8b1e4ad884-kube-api-access-296ck\" (UniqueName: \"kubernetes.io/projected/90acbd64-4882-4fce-92cc-7e8b1e4ad884-kube-api-access-296ck\") pod \"90acbd64-4882-4fce-92cc-7e8b1e4ad884\" (UID: \"90acbd64-4882-4fce-92cc-7e8b1e4ad884\") "
Oct 30 20:47:29.317083 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:29.317054    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90acbd64-4882-4fce-92cc-7e8b1e4ad884-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90acbd64-4882-4fce-92cc-7e8b1e4ad884-maas-api-openshift-ca\") pod \"90acbd64-4882-4fce-92cc-7e8b1e4ad884\" (UID: \"90acbd64-4882-4fce-92cc-7e8b1e4ad884\") "
Oct 30 20:47:29.317423 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:29.317390    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90acbd64-4882-4fce-92cc-7e8b1e4ad884-maas-api-openshift-ca" pod "90acbd64-4882-4fce-92cc-7e8b1e4ad884" (UID: "90acbd64-4882-4fce-92cc-7e8b1e4ad884"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:47:29.317689 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:29.317670    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90acbd64-4882-4fce-92cc-7e8b1e4ad884-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:29.319259 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:29.319238    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90acbd64-4882-4fce-92cc-7e8b1e4ad884-maas-api-xks-ca" pod "90acbd64-4882-4fce-92cc-7e8b1e4ad884" (UID: "90acbd64-4882-4fce-92cc-7e8b1e4ad884"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:47:29.319376 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:29.319357    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90acbd64-4882-4fce-92cc-7e8b1e4ad884-kube-api-access-296ck" pod "90acbd64-4882-4fce-92cc-7e8b1e4ad884" (UID: "90acbd64-4882-4fce-92cc-7e8b1e4ad884"). InnerVolumeSpecName "kube-api-access-296ck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:29.418546 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:29.418507    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90acbd64-4882-4fce-92cc-7e8b1e4ad884-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:29.418546 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:29.418536    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-296ck\" (UniqueName: \"kubernetes.io/projected/90acbd64-4882-4fce-92cc-7e8b1e4ad884-kube-api-access-296ck\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:30.063986 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.063955    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1318e0ff-jwxgh"]
Oct 30 20:47:30.069880 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.069848    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1318e0ff-jwxgh"]
Oct 30 20:47:30.086515 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.086485    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-066357ca-mb8dn"
Oct 30 20:47:30.086673 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.086489    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-066357ca-mb8dn" event={"ID":"90acbd64-4882-4fce-92cc-7e8b1e4ad884","Type":"ContainerDied","Data":"65c3de861617a2a35ed956acad9710f67b31f2918e7c9c60e734c0c4e1bca174"}
Oct 30 20:47:30.086725 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.086682    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65c3de861617a2a35ed956acad9710f67b31f2918e7c9c60e734c0c4e1bca174"
Oct 30 20:47:30.112325 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.112297    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f10c49ba-prq2w"]
Oct 30 20:47:30.112964 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.112950    5071 cpu_manager.go:414] "containerMap: removing container" podUID="90acbd64-4882-4fce-92cc-7e8b1e4ad884" containerName="revoke-keys"
Oct 30 20:47:30.113018 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.112966    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90acbd64-4882-4fce-92cc-7e8b1e4ad884" containerName="revoke-keys"
Oct 30 20:47:30.113084 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.113075    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="90acbd64-4882-4fce-92cc-7e8b1e4ad884" containerName="revoke-keys"
Oct 30 20:47:30.116740 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.116722    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10c49ba-prq2w"
Oct 30 20:47:30.127834 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.127794    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a0683e4-8c11-4ab5-a025-24f1e93b4d01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f10c49ba-prq2w\" (UID: \"9a0683e4-8c11-4ab5-a025-24f1e93b4d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10c49ba-prq2w"
Oct 30 20:47:30.127949 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.127875    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a0683e4-8c11-4ab5-a025-24f1e93b4d01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f10c49ba-prq2w\" (UID: \"9a0683e4-8c11-4ab5-a025-24f1e93b4d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10c49ba-prq2w"
Oct 30 20:47:30.128091 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.128066    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqkj7\" (UniqueName: \"kubernetes.io/projected/9a0683e4-8c11-4ab5-a025-24f1e93b4d01-kube-api-access-zqkj7\") pod \"maas-api-revoke-sub-f10c49ba-prq2w\" (UID: \"9a0683e4-8c11-4ab5-a025-24f1e93b4d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10c49ba-prq2w"
Oct 30 20:47:30.170582 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.170550    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:47:30.170769 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.170758    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 20:47:30.176774 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.176744    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f10c49ba-prq2w"]
Oct 30 20:47:30.206724 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.206692    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4036a76d-bdb3-4542-aae7-50d1bb377050" path="/var/lib/kubelet/pods/4036a76d-bdb3-4542-aae7-50d1bb377050/volumes"
Oct 30 20:47:30.229019 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.228985    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a0683e4-8c11-4ab5-a025-24f1e93b4d01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f10c49ba-prq2w\" (UID: \"9a0683e4-8c11-4ab5-a025-24f1e93b4d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10c49ba-prq2w"
Oct 30 20:47:30.229173 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.229042    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a0683e4-8c11-4ab5-a025-24f1e93b4d01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f10c49ba-prq2w\" (UID: \"9a0683e4-8c11-4ab5-a025-24f1e93b4d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10c49ba-prq2w"
Oct 30 20:47:30.229240 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.229200    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqkj7\" (UniqueName: \"kubernetes.io/projected/9a0683e4-8c11-4ab5-a025-24f1e93b4d01-kube-api-access-zqkj7\") pod \"maas-api-revoke-sub-f10c49ba-prq2w\" (UID: \"9a0683e4-8c11-4ab5-a025-24f1e93b4d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10c49ba-prq2w"
Oct 30 20:47:30.229749 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.229724    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a0683e4-8c11-4ab5-a025-24f1e93b4d01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f10c49ba-prq2w\" (UID: \"9a0683e4-8c11-4ab5-a025-24f1e93b4d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10c49ba-prq2w"
Oct 30 20:47:30.231594 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.231573    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a0683e4-8c11-4ab5-a025-24f1e93b4d01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f10c49ba-prq2w\" (UID: \"9a0683e4-8c11-4ab5-a025-24f1e93b4d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10c49ba-prq2w"
Oct 30 20:47:30.289985 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.289952    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqkj7\" (UniqueName: \"kubernetes.io/projected/9a0683e4-8c11-4ab5-a025-24f1e93b4d01-kube-api-access-zqkj7\") pod \"maas-api-revoke-sub-f10c49ba-prq2w\" (UID: \"9a0683e4-8c11-4ab5-a025-24f1e93b4d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10c49ba-prq2w"
Oct 30 20:47:30.431521 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.431446    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10c49ba-prq2w"
Oct 30 20:47:30.500134 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.500082    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:30.675326 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.675297    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"]
Oct 30 20:47:30.680909 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.680883    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"
Oct 30 20:47:30.736592 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.736558    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vml6q\" (UniqueName: \"kubernetes.io/projected/5cdad370-2ff5-435c-bbfe-2ba61a325d47-kube-api-access-vml6q\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb\" (UID: \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"
Oct 30 20:47:30.736768 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.736611    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cdad370-2ff5-435c-bbfe-2ba61a325d47-credential-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb\" (UID: \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"
Oct 30 20:47:30.736768 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.736644    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cdad370-2ff5-435c-bbfe-2ba61a325d47-istio-envoy\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb\" (UID: \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"
Oct 30 20:47:30.736768 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.736685    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cdad370-2ff5-435c-bbfe-2ba61a325d47-workload-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb\" (UID: \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"
Oct 30 20:47:30.736768 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.736706    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cdad370-2ff5-435c-bbfe-2ba61a325d47-istiod-ca-cert\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb\" (UID: \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"
Oct 30 20:47:30.736768 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.736731    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cdad370-2ff5-435c-bbfe-2ba61a325d47-istio-data\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb\" (UID: \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"
Oct 30 20:47:30.737053 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.736779    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cdad370-2ff5-435c-bbfe-2ba61a325d47-istio-token\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb\" (UID: \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"
Oct 30 20:47:30.737053 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.736842    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cdad370-2ff5-435c-bbfe-2ba61a325d47-workload-certs\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb\" (UID: \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"
Oct 30 20:47:30.737053 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.736871    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cdad370-2ff5-435c-bbfe-2ba61a325d47-istio-podinfo\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb\" (UID: \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"
Oct 30 20:47:30.756112 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.756081    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"]
Oct 30 20:47:30.815072 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:47:30.815040    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a0683e4_8c11_4ab5_a025_24f1e93b4d01.slice/crio-aba7c5b0f8d044846ddd6ce902c6f0f893d67ef83b7abd61971b112481ae47c3 WatchSource:0}: Error finding container aba7c5b0f8d044846ddd6ce902c6f0f893d67ef83b7abd61971b112481ae47c3: Status 404 returned error can't find the container with id aba7c5b0f8d044846ddd6ce902c6f0f893d67ef83b7abd61971b112481ae47c3
Oct 30 20:47:30.816127 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.816105    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f10c49ba-prq2w"]
Oct 30 20:47:30.837943 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.837798    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vml6q\" (UniqueName: \"kubernetes.io/projected/5cdad370-2ff5-435c-bbfe-2ba61a325d47-kube-api-access-vml6q\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb\" (UID: \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"
Oct 30 20:47:30.837943 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.837893    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cdad370-2ff5-435c-bbfe-2ba61a325d47-credential-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb\" (UID: \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"
Oct 30 20:47:30.837943 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.837928    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cdad370-2ff5-435c-bbfe-2ba61a325d47-istio-envoy\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb\" (UID: \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"
Oct 30 20:47:30.838209 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.837969    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cdad370-2ff5-435c-bbfe-2ba61a325d47-workload-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb\" (UID: \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"
Oct 30 20:47:30.838209 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.837990    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cdad370-2ff5-435c-bbfe-2ba61a325d47-istiod-ca-cert\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb\" (UID: \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"
Oct 30 20:47:30.838209 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.838028    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cdad370-2ff5-435c-bbfe-2ba61a325d47-istio-data\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb\" (UID: \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"
Oct 30 20:47:30.838209 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.838083    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cdad370-2ff5-435c-bbfe-2ba61a325d47-istio-token\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb\" (UID: \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"
Oct 30 20:47:30.838209 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.838130    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cdad370-2ff5-435c-bbfe-2ba61a325d47-workload-certs\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb\" (UID: \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"
Oct 30 20:47:30.838209 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.838162    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cdad370-2ff5-435c-bbfe-2ba61a325d47-istio-podinfo\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb\" (UID: \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"
Oct 30 20:47:30.839250 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.839214    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cdad370-2ff5-435c-bbfe-2ba61a325d47-workload-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb\" (UID: \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"
Oct 30 20:47:30.839741 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.839716    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cdad370-2ff5-435c-bbfe-2ba61a325d47-credential-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb\" (UID: \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"
Oct 30 20:47:30.850178 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.841132    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cdad370-2ff5-435c-bbfe-2ba61a325d47-workload-certs\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb\" (UID: \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"
Oct 30 20:47:30.850178 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.841805    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cdad370-2ff5-435c-bbfe-2ba61a325d47-istiod-ca-cert\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb\" (UID: \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"
Oct 30 20:47:30.850178 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.842145    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cdad370-2ff5-435c-bbfe-2ba61a325d47-istio-data\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb\" (UID: \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"
Oct 30 20:47:30.862918 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.859803    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:30.879629 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.878581    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cdad370-2ff5-435c-bbfe-2ba61a325d47-istio-podinfo\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb\" (UID: \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"
Oct 30 20:47:30.904894 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.904806    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vml6q\" (UniqueName: \"kubernetes.io/projected/5cdad370-2ff5-435c-bbfe-2ba61a325d47-kube-api-access-vml6q\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb\" (UID: \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"
Oct 30 20:47:30.908375 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.907121    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cdad370-2ff5-435c-bbfe-2ba61a325d47-istio-envoy\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb\" (UID: \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"
Oct 30 20:47:30.951896 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.930384    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cdad370-2ff5-435c-bbfe-2ba61a325d47-istio-token\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb\" (UID: \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"
Oct 30 20:47:30.997227 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:30.997198    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"
Oct 30 20:47:31.098152 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.098100    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10c49ba-prq2w" event={"ID":"9a0683e4-8c11-4ab5-a025-24f1e93b4d01","Type":"ContainerStarted","Data":"aba7c5b0f8d044846ddd6ce902c6f0f893d67ef83b7abd61971b112481ae47c3"}
Oct 30 20:47:31.372881 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.371663    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"]
Oct 30 20:47:31.382408 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.382382    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"
Oct 30 20:47:31.427037 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.427007    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"]
Oct 30 20:47:31.449545 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.448341    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"]
Oct 30 20:47:31.452040 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.452010    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80fd141-e109-4216-a3f1-39d326d72ab1-credential-socket\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9\" (UID: \"c80fd141-e109-4216-a3f1-39d326d72ab1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"
Oct 30 20:47:31.452226 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.452192    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c80fd141-e109-4216-a3f1-39d326d72ab1-istio-envoy\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9\" (UID: \"c80fd141-e109-4216-a3f1-39d326d72ab1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"
Oct 30 20:47:31.452351 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.452301    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdlfz\" (UniqueName: \"kubernetes.io/projected/c80fd141-e109-4216-a3f1-39d326d72ab1-kube-api-access-tdlfz\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9\" (UID: \"c80fd141-e109-4216-a3f1-39d326d72ab1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"
Oct 30 20:47:31.452436 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.452387    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c80fd141-e109-4216-a3f1-39d326d72ab1-istio-data\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9\" (UID: \"c80fd141-e109-4216-a3f1-39d326d72ab1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"
Oct 30 20:47:31.452529 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.452476    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c80fd141-e109-4216-a3f1-39d326d72ab1-istiod-ca-cert\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9\" (UID: \"c80fd141-e109-4216-a3f1-39d326d72ab1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"
Oct 30 20:47:31.452592 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.452547    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c80fd141-e109-4216-a3f1-39d326d72ab1-istio-token\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9\" (UID: \"c80fd141-e109-4216-a3f1-39d326d72ab1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"
Oct 30 20:47:31.452695 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.452678    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80fd141-e109-4216-a3f1-39d326d72ab1-workload-socket\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9\" (UID: \"c80fd141-e109-4216-a3f1-39d326d72ab1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"
Oct 30 20:47:31.452968 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.452946    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c80fd141-e109-4216-a3f1-39d326d72ab1-workload-certs\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9\" (UID: \"c80fd141-e109-4216-a3f1-39d326d72ab1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"
Oct 30 20:47:31.453115 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.453019    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c80fd141-e109-4216-a3f1-39d326d72ab1-istio-podinfo\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9\" (UID: \"c80fd141-e109-4216-a3f1-39d326d72ab1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"
Oct 30 20:47:31.458963 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.458931    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:47:31.459158 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.459134    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:47:31.459283 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.459263    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:47:31.554154 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.554115    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c80fd141-e109-4216-a3f1-39d326d72ab1-workload-certs\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9\" (UID: \"c80fd141-e109-4216-a3f1-39d326d72ab1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"
Oct 30 20:47:31.554571 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.554166    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c80fd141-e109-4216-a3f1-39d326d72ab1-istio-podinfo\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9\" (UID: \"c80fd141-e109-4216-a3f1-39d326d72ab1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"
Oct 30 20:47:31.554571 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.554213    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80fd141-e109-4216-a3f1-39d326d72ab1-credential-socket\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9\" (UID: \"c80fd141-e109-4216-a3f1-39d326d72ab1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"
Oct 30 20:47:31.554571 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.554265    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c80fd141-e109-4216-a3f1-39d326d72ab1-istio-envoy\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9\" (UID: \"c80fd141-e109-4216-a3f1-39d326d72ab1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"
Oct 30 20:47:31.554571 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.554320    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdlfz\" (UniqueName: \"kubernetes.io/projected/c80fd141-e109-4216-a3f1-39d326d72ab1-kube-api-access-tdlfz\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9\" (UID: \"c80fd141-e109-4216-a3f1-39d326d72ab1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"
Oct 30 20:47:31.554571 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.554360    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c80fd141-e109-4216-a3f1-39d326d72ab1-istio-data\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9\" (UID: \"c80fd141-e109-4216-a3f1-39d326d72ab1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"
Oct 30 20:47:31.554571 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.554387    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c80fd141-e109-4216-a3f1-39d326d72ab1-istiod-ca-cert\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9\" (UID: \"c80fd141-e109-4216-a3f1-39d326d72ab1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"
Oct 30 20:47:31.554571 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.554426    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c80fd141-e109-4216-a3f1-39d326d72ab1-istio-token\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9\" (UID: \"c80fd141-e109-4216-a3f1-39d326d72ab1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"
Oct 30 20:47:31.554571 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.554493    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80fd141-e109-4216-a3f1-39d326d72ab1-workload-socket\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9\" (UID: \"c80fd141-e109-4216-a3f1-39d326d72ab1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"
Oct 30 20:47:31.554571 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.554561    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c80fd141-e109-4216-a3f1-39d326d72ab1-workload-certs\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9\" (UID: \"c80fd141-e109-4216-a3f1-39d326d72ab1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"
Oct 30 20:47:31.555071 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.554618    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80fd141-e109-4216-a3f1-39d326d72ab1-credential-socket\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9\" (UID: \"c80fd141-e109-4216-a3f1-39d326d72ab1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"
Oct 30 20:47:31.555138 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.555078    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c80fd141-e109-4216-a3f1-39d326d72ab1-istio-data\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9\" (UID: \"c80fd141-e109-4216-a3f1-39d326d72ab1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"
Oct 30 20:47:31.555428 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.555400    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80fd141-e109-4216-a3f1-39d326d72ab1-workload-socket\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9\" (UID: \"c80fd141-e109-4216-a3f1-39d326d72ab1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"
Oct 30 20:47:31.555428 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.555416    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c80fd141-e109-4216-a3f1-39d326d72ab1-istiod-ca-cert\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9\" (UID: \"c80fd141-e109-4216-a3f1-39d326d72ab1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"
Oct 30 20:47:31.558230 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.558182    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c80fd141-e109-4216-a3f1-39d326d72ab1-istio-podinfo\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9\" (UID: \"c80fd141-e109-4216-a3f1-39d326d72ab1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"
Oct 30 20:47:31.569171 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.568727    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c80fd141-e109-4216-a3f1-39d326d72ab1-istio-envoy\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9\" (UID: \"c80fd141-e109-4216-a3f1-39d326d72ab1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"
Oct 30 20:47:31.617041 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.617004    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c80fd141-e109-4216-a3f1-39d326d72ab1-istio-token\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9\" (UID: \"c80fd141-e109-4216-a3f1-39d326d72ab1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"
Oct 30 20:47:31.618340 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.618313    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdlfz\" (UniqueName: \"kubernetes.io/projected/c80fd141-e109-4216-a3f1-39d326d72ab1-kube-api-access-tdlfz\") pod \"e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9\" (UID: \"c80fd141-e109-4216-a3f1-39d326d72ab1\") " pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"
Oct 30 20:47:31.697931 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.697894    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"
Oct 30 20:47:31.801127 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.801094    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"]
Oct 30 20:47:31.807985 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.807964    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"
Oct 30 20:47:31.848437 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.848398    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"]
Oct 30 20:47:31.854336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.854316    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"
Oct 30 20:47:31.881583 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.881549    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6295120b-8bbc-4502-8429-4f04d7d5c5a4-workload-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk\" (UID: \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"
Oct 30 20:47:31.881898 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.881814    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d634d9-39de-4614-9599-38adf86829e2-workload-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42\" (UID: \"c4d634d9-39de-4614-9599-38adf86829e2\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"
Oct 30 20:47:31.882060 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.882041    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2hcg\" (UniqueName: \"kubernetes.io/projected/c4d634d9-39de-4614-9599-38adf86829e2-kube-api-access-x2hcg\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42\" (UID: \"c4d634d9-39de-4614-9599-38adf86829e2\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"
Oct 30 20:47:31.882227 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.882208    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6295120b-8bbc-4502-8429-4f04d7d5c5a4-istio-podinfo\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk\" (UID: \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"
Oct 30 20:47:31.882426 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.882407    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d634d9-39de-4614-9599-38adf86829e2-credential-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42\" (UID: \"c4d634d9-39de-4614-9599-38adf86829e2\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"
Oct 30 20:47:31.882663 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.882635    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4d634d9-39de-4614-9599-38adf86829e2-istio-podinfo\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42\" (UID: \"c4d634d9-39de-4614-9599-38adf86829e2\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"
Oct 30 20:47:31.882926 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.882897    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4d634d9-39de-4614-9599-38adf86829e2-istio-envoy\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42\" (UID: \"c4d634d9-39de-4614-9599-38adf86829e2\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"
Oct 30 20:47:31.883205 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.883186    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4d634d9-39de-4614-9599-38adf86829e2-workload-certs\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42\" (UID: \"c4d634d9-39de-4614-9599-38adf86829e2\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"
Oct 30 20:47:31.883429 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.883412    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4d634d9-39de-4614-9599-38adf86829e2-istiod-ca-cert\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42\" (UID: \"c4d634d9-39de-4614-9599-38adf86829e2\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"
Oct 30 20:47:31.883654 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.883637    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6295120b-8bbc-4502-8429-4f04d7d5c5a4-istio-token\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk\" (UID: \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"
Oct 30 20:47:31.883864 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.883847    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4d634d9-39de-4614-9599-38adf86829e2-istio-data\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42\" (UID: \"c4d634d9-39de-4614-9599-38adf86829e2\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"
Oct 30 20:47:31.884042 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.884026    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4d634d9-39de-4614-9599-38adf86829e2-istio-token\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42\" (UID: \"c4d634d9-39de-4614-9599-38adf86829e2\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"
Oct 30 20:47:31.884211 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.884196    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6295120b-8bbc-4502-8429-4f04d7d5c5a4-credential-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk\" (UID: \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"
Oct 30 20:47:31.884372 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.884356    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c82rd\" (UniqueName: \"kubernetes.io/projected/6295120b-8bbc-4502-8429-4f04d7d5c5a4-kube-api-access-c82rd\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk\" (UID: \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"
Oct 30 20:47:31.884544 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.884529    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6295120b-8bbc-4502-8429-4f04d7d5c5a4-workload-certs\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk\" (UID: \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"
Oct 30 20:47:31.884711 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.884696    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6295120b-8bbc-4502-8429-4f04d7d5c5a4-istio-data\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk\" (UID: \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"
Oct 30 20:47:31.884955 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.884936    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6295120b-8bbc-4502-8429-4f04d7d5c5a4-istio-envoy\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk\" (UID: \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"
Oct 30 20:47:31.885097 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.885083    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6295120b-8bbc-4502-8429-4f04d7d5c5a4-istiod-ca-cert\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk\" (UID: \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"
Oct 30 20:47:31.917323 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.917295    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"]
Oct 30 20:47:31.923270 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.923248    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"]
Oct 30 20:47:31.986285 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.986254    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4d634d9-39de-4614-9599-38adf86829e2-istio-podinfo\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42\" (UID: \"c4d634d9-39de-4614-9599-38adf86829e2\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"
Oct 30 20:47:31.986464 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.986304    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4d634d9-39de-4614-9599-38adf86829e2-istio-envoy\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42\" (UID: \"c4d634d9-39de-4614-9599-38adf86829e2\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"
Oct 30 20:47:31.986464 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.986333    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4d634d9-39de-4614-9599-38adf86829e2-workload-certs\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42\" (UID: \"c4d634d9-39de-4614-9599-38adf86829e2\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"
Oct 30 20:47:31.986464 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.986361    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4d634d9-39de-4614-9599-38adf86829e2-istiod-ca-cert\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42\" (UID: \"c4d634d9-39de-4614-9599-38adf86829e2\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"
Oct 30 20:47:31.986464 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.986398    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6295120b-8bbc-4502-8429-4f04d7d5c5a4-istio-token\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk\" (UID: \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"
Oct 30 20:47:31.986464 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.986425    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4d634d9-39de-4614-9599-38adf86829e2-istio-data\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42\" (UID: \"c4d634d9-39de-4614-9599-38adf86829e2\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"
Oct 30 20:47:31.986464 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.986441    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4d634d9-39de-4614-9599-38adf86829e2-istio-token\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42\" (UID: \"c4d634d9-39de-4614-9599-38adf86829e2\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"
Oct 30 20:47:31.986464 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.986462    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6295120b-8bbc-4502-8429-4f04d7d5c5a4-credential-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk\" (UID: \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"
Oct 30 20:47:31.986896 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.986480    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c82rd\" (UniqueName: \"kubernetes.io/projected/6295120b-8bbc-4502-8429-4f04d7d5c5a4-kube-api-access-c82rd\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk\" (UID: \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"
Oct 30 20:47:31.986896 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.986514    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6295120b-8bbc-4502-8429-4f04d7d5c5a4-workload-certs\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk\" (UID: \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"
Oct 30 20:47:31.986896 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.986534    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6295120b-8bbc-4502-8429-4f04d7d5c5a4-istio-data\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk\" (UID: \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"
Oct 30 20:47:31.986896 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.986586    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6295120b-8bbc-4502-8429-4f04d7d5c5a4-istio-envoy\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk\" (UID: \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"
Oct 30 20:47:31.986896 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.986608    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6295120b-8bbc-4502-8429-4f04d7d5c5a4-istiod-ca-cert\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk\" (UID: \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"
Oct 30 20:47:31.986896 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.986631    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6295120b-8bbc-4502-8429-4f04d7d5c5a4-workload-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk\" (UID: \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"
Oct 30 20:47:31.986896 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.986669    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d634d9-39de-4614-9599-38adf86829e2-workload-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42\" (UID: \"c4d634d9-39de-4614-9599-38adf86829e2\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"
Oct 30 20:47:31.986896 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.986690    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2hcg\" (UniqueName: \"kubernetes.io/projected/c4d634d9-39de-4614-9599-38adf86829e2-kube-api-access-x2hcg\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42\" (UID: \"c4d634d9-39de-4614-9599-38adf86829e2\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"
Oct 30 20:47:31.986896 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.986711    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6295120b-8bbc-4502-8429-4f04d7d5c5a4-istio-podinfo\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk\" (UID: \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"
Oct 30 20:47:31.986896 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.986731    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d634d9-39de-4614-9599-38adf86829e2-credential-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42\" (UID: \"c4d634d9-39de-4614-9599-38adf86829e2\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"
Oct 30 20:47:31.987383 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.987077    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d634d9-39de-4614-9599-38adf86829e2-credential-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42\" (UID: \"c4d634d9-39de-4614-9599-38adf86829e2\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"
Oct 30 20:47:31.988732 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.987785    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d634d9-39de-4614-9599-38adf86829e2-workload-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42\" (UID: \"c4d634d9-39de-4614-9599-38adf86829e2\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"
Oct 30 20:47:31.988732 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.987924    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6295120b-8bbc-4502-8429-4f04d7d5c5a4-istio-data\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk\" (UID: \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"
Oct 30 20:47:31.988732 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.988077    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6295120b-8bbc-4502-8429-4f04d7d5c5a4-workload-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk\" (UID: \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"
Oct 30 20:47:31.988732 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.988175    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6295120b-8bbc-4502-8429-4f04d7d5c5a4-workload-certs\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk\" (UID: \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"
Oct 30 20:47:31.988732 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.988334    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6295120b-8bbc-4502-8429-4f04d7d5c5a4-istiod-ca-cert\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk\" (UID: \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"
Oct 30 20:47:31.989236 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.988757    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4d634d9-39de-4614-9599-38adf86829e2-istio-data\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42\" (UID: \"c4d634d9-39de-4614-9599-38adf86829e2\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"
Oct 30 20:47:31.989328 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.989307    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4d634d9-39de-4614-9599-38adf86829e2-workload-certs\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42\" (UID: \"c4d634d9-39de-4614-9599-38adf86829e2\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"
Oct 30 20:47:31.989452 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.989406    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4d634d9-39de-4614-9599-38adf86829e2-istiod-ca-cert\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42\" (UID: \"c4d634d9-39de-4614-9599-38adf86829e2\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"
Oct 30 20:47:31.989588 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.989566    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6295120b-8bbc-4502-8429-4f04d7d5c5a4-credential-socket\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk\" (UID: \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"
Oct 30 20:47:31.992371 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.992347    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6295120b-8bbc-4502-8429-4f04d7d5c5a4-istio-envoy\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk\" (UID: \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"
Oct 30 20:47:31.992371 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.992363    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6295120b-8bbc-4502-8429-4f04d7d5c5a4-istio-podinfo\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk\" (UID: \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"
Oct 30 20:47:31.992538 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.992479    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4d634d9-39de-4614-9599-38adf86829e2-istio-podinfo\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42\" (UID: \"c4d634d9-39de-4614-9599-38adf86829e2\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"
Oct 30 20:47:31.993989 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:31.993970    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4d634d9-39de-4614-9599-38adf86829e2-istio-envoy\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42\" (UID: \"c4d634d9-39de-4614-9599-38adf86829e2\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"
Oct 30 20:47:32.047302 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.047247    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4d634d9-39de-4614-9599-38adf86829e2-istio-token\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42\" (UID: \"c4d634d9-39de-4614-9599-38adf86829e2\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"
Oct 30 20:47:32.047607 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.047562    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c82rd\" (UniqueName: \"kubernetes.io/projected/6295120b-8bbc-4502-8429-4f04d7d5c5a4-kube-api-access-c82rd\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk\" (UID: \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"
Oct 30 20:47:32.048934 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.048908    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6295120b-8bbc-4502-8429-4f04d7d5c5a4-istio-token\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk\" (UID: \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"
Oct 30 20:47:32.049194 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.049163    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2hcg\" (UniqueName: \"kubernetes.io/projected/c4d634d9-39de-4614-9599-38adf86829e2-kube-api-access-x2hcg\") pod \"e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42\" (UID: \"c4d634d9-39de-4614-9599-38adf86829e2\") " pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"
Oct 30 20:47:32.065572 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.065542    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dd79b45-sshj7"]
Oct 30 20:47:32.069709 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.069684    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dd79b45-sshj7"]
Oct 30 20:47:32.107147 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.107117    5071 generic.go:356] "Generic (PLEG): container finished" podID="9a0683e4-8c11-4ab5-a025-24f1e93b4d01" containerID="a64ab1a5aafdc217552106d41bd42ffbf5e5dde053859890050e04011ce7663f" exitCode=0
Oct 30 20:47:32.107273 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.107213    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10c49ba-prq2w" event={"ID":"9a0683e4-8c11-4ab5-a025-24f1e93b4d01","Type":"ContainerDied","Data":"a64ab1a5aafdc217552106d41bd42ffbf5e5dde053859890050e04011ce7663f"}
Oct 30 20:47:32.109284 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.109262    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb" event={"ID":"5cdad370-2ff5-435c-bbfe-2ba61a325d47","Type":"ContainerStarted","Data":"22a7dc2536154ac2157046ebc9b68adc172c85397eb9352cfc893e481db56048"}
Oct 30 20:47:32.109405 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.109302    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb" event={"ID":"5cdad370-2ff5-435c-bbfe-2ba61a325d47","Type":"ContainerStarted","Data":"064e1ca3ff6e361bb67ba4354e3c41c2480b015c51792f410c6253b159b42d3d"}
Oct 30 20:47:32.129021 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.128992    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"
Oct 30 20:47:32.141669 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.141638    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"]
Oct 30 20:47:32.144670 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:47:32.144645    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc80fd141_e109_4216_a3f1_39d326d72ab1.slice/crio-49b4c09795a48297c67f8d5e79d80d6e23683473a08db69aeeb2c69d0171f27e WatchSource:0}: Error finding container 49b4c09795a48297c67f8d5e79d80d6e23683473a08db69aeeb2c69d0171f27e: Status 404 returned error can't find the container with id 49b4c09795a48297c67f8d5e79d80d6e23683473a08db69aeeb2c69d0171f27e
Oct 30 20:47:32.149031 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.148989    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:47:32.149106 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.149078    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:47:32.149154 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.149120    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:47:32.199662 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.199518    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"
Oct 30 20:47:32.210752 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.210694    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7cd5a70-c322-4277-8a96-ebb7b539d088" path="/var/lib/kubelet/pods/d7cd5a70-c322-4277-8a96-ebb7b539d088/volumes"
Oct 30 20:47:32.360113 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.360054    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb" podStartSLOduration=2.360034102 podStartE2EDuration="2.360034102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:32.353497463 +0000 UTC m=+1264.270519052" watchObservedRunningTime="2026-10-30 20:47:32.360034102 +0000 UTC m=+1264.277055640"
Oct 30 20:47:32.582956 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.580302    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:47:32.582956 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.580400    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:47:32.582956 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.580446    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:47:32.584404 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.584376    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"]
Oct 30 20:47:32.675103 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.675075    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"]
Oct 30 20:47:32.678573 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.678536    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:47:32.678704 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.678618    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:47:32.678704 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.678669    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:47:32.751762 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.751726    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38cb58e4-bpnf4"]
Oct 30 20:47:32.777272 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.777231    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38cb58e4-bpnf4"
Oct 30 20:47:32.799984 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.799953    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38cb58e4-bpnf4"]
Oct 30 20:47:32.800355 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.799968    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3218c9e-b073-4137-9c34-26f2d0294ad0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38cb58e4-bpnf4\" (UID: \"a3218c9e-b073-4137-9c34-26f2d0294ad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38cb58e4-bpnf4"
Oct 30 20:47:32.800597 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.800560    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3218c9e-b073-4137-9c34-26f2d0294ad0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38cb58e4-bpnf4\" (UID: \"a3218c9e-b073-4137-9c34-26f2d0294ad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38cb58e4-bpnf4"
Oct 30 20:47:32.800859 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.800814    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mf45\" (UniqueName: \"kubernetes.io/projected/a3218c9e-b073-4137-9c34-26f2d0294ad0-kube-api-access-7mf45\") pod \"maas-api-revoke-sub-38cb58e4-bpnf4\" (UID: \"a3218c9e-b073-4137-9c34-26f2d0294ad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38cb58e4-bpnf4"
Oct 30 20:47:32.923238 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.921028    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3218c9e-b073-4137-9c34-26f2d0294ad0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38cb58e4-bpnf4\" (UID: \"a3218c9e-b073-4137-9c34-26f2d0294ad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38cb58e4-bpnf4"
Oct 30 20:47:32.923238 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.921232    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mf45\" (UniqueName: \"kubernetes.io/projected/a3218c9e-b073-4137-9c34-26f2d0294ad0-kube-api-access-7mf45\") pod \"maas-api-revoke-sub-38cb58e4-bpnf4\" (UID: \"a3218c9e-b073-4137-9c34-26f2d0294ad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38cb58e4-bpnf4"
Oct 30 20:47:32.923238 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.921497    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3218c9e-b073-4137-9c34-26f2d0294ad0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38cb58e4-bpnf4\" (UID: \"a3218c9e-b073-4137-9c34-26f2d0294ad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38cb58e4-bpnf4"
Oct 30 20:47:32.923238 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.922515    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3218c9e-b073-4137-9c34-26f2d0294ad0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38cb58e4-bpnf4\" (UID: \"a3218c9e-b073-4137-9c34-26f2d0294ad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38cb58e4-bpnf4"
Oct 30 20:47:32.925888 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.925677    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3218c9e-b073-4137-9c34-26f2d0294ad0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38cb58e4-bpnf4\" (UID: \"a3218c9e-b073-4137-9c34-26f2d0294ad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38cb58e4-bpnf4"
Oct 30 20:47:32.992465 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.992428    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mf45\" (UniqueName: \"kubernetes.io/projected/a3218c9e-b073-4137-9c34-26f2d0294ad0-kube-api-access-7mf45\") pod \"maas-api-revoke-sub-38cb58e4-bpnf4\" (UID: \"a3218c9e-b073-4137-9c34-26f2d0294ad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38cb58e4-bpnf4"
Oct 30 20:47:32.997916 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:32.997890    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"
Oct 30 20:47:33.000255 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:33.000235    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 20:47:33.000422 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:33.000397    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb" podUID="5cdad370-2ff5-435c-bbfe-2ba61a325d47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 20:47:33.115927 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:33.115183    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38cb58e4-bpnf4"
Oct 30 20:47:33.124016 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:33.123978    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42" event={"ID":"c4d634d9-39de-4614-9599-38adf86829e2","Type":"ContainerStarted","Data":"ccecb5607c37595ea3fe07271e177ea3db15296bbeecbed3def11b721934dda7"}
Oct 30 20:47:33.124246 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:33.124223    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42" event={"ID":"c4d634d9-39de-4614-9599-38adf86829e2","Type":"ContainerStarted","Data":"826c0ee5498498410d9545468f8f6913141cfa5644fac804d48ff3493b8e0ee5"}
Oct 30 20:47:33.129851 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:33.128477    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9" event={"ID":"c80fd141-e109-4216-a3f1-39d326d72ab1","Type":"ContainerStarted","Data":"5d588107127690f85d2e4027d3b78fcf6e4b670f2f33679c83e80fbcdced4d8b"}
Oct 30 20:47:33.129851 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:33.128528    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9" event={"ID":"c80fd141-e109-4216-a3f1-39d326d72ab1","Type":"ContainerStarted","Data":"49b4c09795a48297c67f8d5e79d80d6e23683473a08db69aeeb2c69d0171f27e"}
Oct 30 20:47:33.132497 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:33.132477    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"
Oct 30 20:47:33.132718 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:33.132700    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 20:47:33.132886 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:33.132863    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42" podUID="c4d634d9-39de-4614-9599-38adf86829e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 20:47:33.152177 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:33.152144    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk" event={"ID":"6295120b-8bbc-4502-8429-4f04d7d5c5a4","Type":"ContainerStarted","Data":"45b8103cf080d88ad6ba8310fa33035ccf747e9b9195dc503291a1adba11ccc9"}
Oct 30 20:47:33.152391 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:33.152369    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk" event={"ID":"6295120b-8bbc-4502-8429-4f04d7d5c5a4","Type":"ContainerStarted","Data":"fd8c9cdfd3092aed8afff9f5a59bb6acccf010b0750ba2b3a9d8e5e0071a1be6"}
Oct 30 20:47:33.202610 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:33.201945    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"
Oct 30 20:47:33.202610 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:33.202250    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 20:47:33.202610 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:33.202325    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk" podUID="6295120b-8bbc-4502-8429-4f04d7d5c5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 20:47:33.308844 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:33.293631    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42" podStartSLOduration=2.293609534 podStartE2EDuration="2.293609534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:33.255623624 +0000 UTC m=+1265.172645161" watchObservedRunningTime="2026-10-30 20:47:33.293609534 +0000 UTC m=+1265.210631104"
Oct 30 20:47:33.455901 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:33.451737    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9" podStartSLOduration=2.451717569 podStartE2EDuration="2.451717569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:33.399318703 +0000 UTC m=+1265.316340239" watchObservedRunningTime="2026-10-30 20:47:33.451717569 +0000 UTC m=+1265.368739108"
Oct 30 20:47:33.601664 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:33.601602    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk" podStartSLOduration=2.601579898 podStartE2EDuration="2.601579898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:33.57512558 +0000 UTC m=+1265.492147117" watchObservedRunningTime="2026-10-30 20:47:33.601579898 +0000 UTC m=+1265.518601435"
Oct 30 20:47:33.704604 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:33.703231    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"
Oct 30 20:47:33.704604 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:33.703984    5071 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 20:47:33.704604 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:33.704034    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9" podUID="c80fd141-e109-4216-a3f1-39d326d72ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 20:47:34.097945 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:34.080737    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 20:47:34.097945 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:34.080802    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb" podUID="5cdad370-2ff5-435c-bbfe-2ba61a325d47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 20:47:34.236854 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:34.228919    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 20:47:34.236854 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:34.228982    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42" podUID="c4d634d9-39de-4614-9599-38adf86829e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 20:47:34.418154 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:34.413315    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 20:47:34.418154 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:34.413368    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk" podUID="6295120b-8bbc-4502-8429-4f04d7d5c5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 20:47:34.706524 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:34.706392    5071 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 20:47:34.706524 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:34.706449    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9" podUID="c80fd141-e109-4216-a3f1-39d326d72ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 20:47:34.720632 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:34.720604    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10c49ba-prq2w"
Oct 30 20:47:34.829849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:34.807347    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a0683e4-8c11-4ab5-a025-24f1e93b4d01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a0683e4-8c11-4ab5-a025-24f1e93b4d01-maas-api-xks-ca\") pod \"9a0683e4-8c11-4ab5-a025-24f1e93b4d01\" (UID: \"9a0683e4-8c11-4ab5-a025-24f1e93b4d01\") "
Oct 30 20:47:34.830356 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:34.830327    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a0683e4-8c11-4ab5-a025-24f1e93b4d01-kube-api-access-zqkj7\" (UniqueName: \"kubernetes.io/projected/9a0683e4-8c11-4ab5-a025-24f1e93b4d01-kube-api-access-zqkj7\") pod \"9a0683e4-8c11-4ab5-a025-24f1e93b4d01\" (UID: \"9a0683e4-8c11-4ab5-a025-24f1e93b4d01\") "
Oct 30 20:47:34.981901 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:34.886890    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 20:47:34.981901 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:34.959252    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10c49ba-prq2w"
Oct 30 20:47:34.981901 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:34.960976    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a0683e4-8c11-4ab5-a025-24f1e93b4d01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a0683e4-8c11-4ab5-a025-24f1e93b4d01-maas-api-openshift-ca\") pod \"9a0683e4-8c11-4ab5-a025-24f1e93b4d01\" (UID: \"9a0683e4-8c11-4ab5-a025-24f1e93b4d01\") "
Oct 30 20:47:35.087553 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:34.993097    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a0683e4-8c11-4ab5-a025-24f1e93b4d01-maas-api-openshift-ca" pod "9a0683e4-8c11-4ab5-a025-24f1e93b4d01" (UID: "9a0683e4-8c11-4ab5-a025-24f1e93b4d01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:47:35.087553 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.008996    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 20:47:35.087553 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.009075    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb" podUID="5cdad370-2ff5-435c-bbfe-2ba61a325d47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 20:47:35.087553 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.074085    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38cb58e4-bpnf4"]
Oct 30 20:47:35.087553 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.074163    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f10c49ba-prq2w" event={"ID":"9a0683e4-8c11-4ab5-a025-24f1e93b4d01","Type":"ContainerDied","Data":"aba7c5b0f8d044846ddd6ce902c6f0f893d67ef83b7abd61971b112481ae47c3"}
Oct 30 20:47:35.087553 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.074197    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aba7c5b0f8d044846ddd6ce902c6f0f893d67ef83b7abd61971b112481ae47c3"
Oct 30 20:47:35.087553 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.074211    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38cb58e4-bpnf4" event={"ID":"a3218c9e-b073-4137-9c34-26f2d0294ad0","Type":"ContainerStarted","Data":"87ab5d94a223e130bc85bd3fcc8cbfe283cb110c04f7969b67d4739e292bd83e"}
Oct 30 20:47:35.087553 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.081488    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a0683e4-8c11-4ab5-a025-24f1e93b4d01-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:35.106889 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.104394    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a0683e4-8c11-4ab5-a025-24f1e93b4d01-maas-api-xks-ca" pod "9a0683e4-8c11-4ab5-a025-24f1e93b4d01" (UID: "9a0683e4-8c11-4ab5-a025-24f1e93b4d01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:47:35.106889 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.104459    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c08ec9d-5mlgd"]
Oct 30 20:47:35.106889 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.105535    5071 cpu_manager.go:414] "containerMap: removing container" podUID="9a0683e4-8c11-4ab5-a025-24f1e93b4d01" containerName="revoke-keys"
Oct 30 20:47:35.106889 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.105557    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a0683e4-8c11-4ab5-a025-24f1e93b4d01" containerName="revoke-keys"
Oct 30 20:47:35.106889 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.105786    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a0683e4-8c11-4ab5-a025-24f1e93b4d01" containerName="revoke-keys"
Oct 30 20:47:35.131170 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.131139    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c08ec9d-5mlgd"
Oct 30 20:47:35.153127 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.153094    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 20:47:35.153356 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.153325    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42" podUID="c4d634d9-39de-4614-9599-38adf86829e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 20:47:35.157301 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.157272    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a0683e4-8c11-4ab5-a025-24f1e93b4d01-kube-api-access-zqkj7" pod "9a0683e4-8c11-4ab5-a025-24f1e93b4d01" (UID: "9a0683e4-8c11-4ab5-a025-24f1e93b4d01"). InnerVolumeSpecName "kube-api-access-zqkj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:35.187848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.176310    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c08ec9d-5mlgd"]
Oct 30 20:47:35.190471 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.190437    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26zbt\" (UniqueName: \"kubernetes.io/projected/4fde2656-e68e-4dfe-8bde-7138b60889e1-kube-api-access-26zbt\") pod \"maas-api-revoke-sub-9c08ec9d-5mlgd\" (UID: \"4fde2656-e68e-4dfe-8bde-7138b60889e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c08ec9d-5mlgd"
Oct 30 20:47:35.190687 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.190669    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fde2656-e68e-4dfe-8bde-7138b60889e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c08ec9d-5mlgd\" (UID: \"4fde2656-e68e-4dfe-8bde-7138b60889e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c08ec9d-5mlgd"
Oct 30 20:47:35.191004 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.190987    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fde2656-e68e-4dfe-8bde-7138b60889e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c08ec9d-5mlgd\" (UID: \"4fde2656-e68e-4dfe-8bde-7138b60889e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c08ec9d-5mlgd"
Oct 30 20:47:35.191439 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.191418    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a0683e4-8c11-4ab5-a025-24f1e93b4d01-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:35.191583 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.191567    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqkj7\" (UniqueName: \"kubernetes.io/projected/9a0683e4-8c11-4ab5-a025-24f1e93b4d01-kube-api-access-zqkj7\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:35.205066 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.205039    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 20:47:35.205260 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.205234    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk" podUID="6295120b-8bbc-4502-8429-4f04d7d5c5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 20:47:35.315858 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.314664    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fde2656-e68e-4dfe-8bde-7138b60889e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c08ec9d-5mlgd\" (UID: \"4fde2656-e68e-4dfe-8bde-7138b60889e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c08ec9d-5mlgd"
Oct 30 20:47:35.315858 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.315007    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26zbt\" (UniqueName: \"kubernetes.io/projected/4fde2656-e68e-4dfe-8bde-7138b60889e1-kube-api-access-26zbt\") pod \"maas-api-revoke-sub-9c08ec9d-5mlgd\" (UID: \"4fde2656-e68e-4dfe-8bde-7138b60889e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c08ec9d-5mlgd"
Oct 30 20:47:35.315858 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.315050    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fde2656-e68e-4dfe-8bde-7138b60889e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c08ec9d-5mlgd\" (UID: \"4fde2656-e68e-4dfe-8bde-7138b60889e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c08ec9d-5mlgd"
Oct 30 20:47:35.316754 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.316702    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fde2656-e68e-4dfe-8bde-7138b60889e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c08ec9d-5mlgd\" (UID: \"4fde2656-e68e-4dfe-8bde-7138b60889e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c08ec9d-5mlgd"
Oct 30 20:47:35.359313 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.359248    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fde2656-e68e-4dfe-8bde-7138b60889e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c08ec9d-5mlgd\" (UID: \"4fde2656-e68e-4dfe-8bde-7138b60889e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c08ec9d-5mlgd"
Oct 30 20:47:35.427348 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.426910    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26zbt\" (UniqueName: \"kubernetes.io/projected/4fde2656-e68e-4dfe-8bde-7138b60889e1-kube-api-access-26zbt\") pod \"maas-api-revoke-sub-9c08ec9d-5mlgd\" (UID: \"4fde2656-e68e-4dfe-8bde-7138b60889e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c08ec9d-5mlgd"
Oct 30 20:47:35.427348 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.427001    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:35.576941 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.576745    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c08ec9d-5mlgd"
Oct 30 20:47:35.702854 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.700893    5071 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 20:47:35.702854 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.700948    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9" podUID="c80fd141-e109-4216-a3f1-39d326d72ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 20:47:35.877120 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.877076    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 20:47:35.900801 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.900754    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:35.905408 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.905376    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 20:47:35.933798 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:35.933761    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:47:36.076932 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:36.057167    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 20:47:36.076932 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:36.057229    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb" podUID="5cdad370-2ff5-435c-bbfe-2ba61a325d47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 20:47:36.416923 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:36.403431    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 30 20:47:36.416923 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:36.405193    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 20:47:36.416923 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:36.405253    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk" podUID="6295120b-8bbc-4502-8429-4f04d7d5c5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 20:47:36.416923 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:36.409146    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 20:47:36.416923 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:36.410164    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42" podUID="c4d634d9-39de-4614-9599-38adf86829e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 20:47:36.443033 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:36.435759    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:36.836851 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:36.810958    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:47:37.156902 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:36.835388    5071 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 20:47:37.156902 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:37.156204    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9" podUID="c80fd141-e109-4216-a3f1-39d326d72ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 20:47:37.156902 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:36.909927    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:37.197850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:37.179895    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 20:47:37.197850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:37.179961    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42" podUID="c4d634d9-39de-4614-9599-38adf86829e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 20:47:37.197850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:37.069525    5071 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:37.197850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:37.068928    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 20:47:37.197850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:37.180241    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb" podUID="5cdad370-2ff5-435c-bbfe-2ba61a325d47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 20:47:37.197850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:36.880933    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 20:47:37.197850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:37.181438    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:47:37.250921 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:37.212440    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 20:47:37.250921 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:37.212517    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk" podUID="6295120b-8bbc-4502-8429-4f04d7d5c5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 20:47:37.258854 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:36.910375    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:37.258854 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:37.255992    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:47:37.509090 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:37.441687    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:37.509090 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:37.467855    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 30 20:47:37.509090 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:37.487786    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:47:37.560281 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:37.534356    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:37.707936 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:37.681311    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:47:37.847067 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:37.741970    5071 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 20:47:37.847067 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:37.742028    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9" podUID="c80fd141-e109-4216-a3f1-39d326d72ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 20:47:37.880068 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:37.851911    5071 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded"
Oct 30 20:47:38.094882 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:38.087403    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 20:47:38.094882 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:38.087467    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb" podUID="5cdad370-2ff5-435c-bbfe-2ba61a325d47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 20:47:38.094882 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:38.087877    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38cb58e4-bpnf4" event={"ID":"a3218c9e-b073-4137-9c34-26f2d0294ad0","Type":"ContainerStarted","Data":"5084c31899b706da6dea4857528ce6728bbd5b9030660b4652b22645fcb2419f"}
Oct 30 20:47:38.122884 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:38.116933    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c08ec9d-5mlgd"]
Oct 30 20:47:38.215043 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:38.215001    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:38.241895 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:38.216895    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:47:38.241895 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:38.216984    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 20:47:38.241895 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:38.217019    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42" podUID="c4d634d9-39de-4614-9599-38adf86829e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 20:47:38.242739 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:38.242709    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 20:47:38.242947 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:38.242921    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk" podUID="6295120b-8bbc-4502-8429-4f04d7d5c5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 20:47:38.259964 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:38.259908    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:38.486848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:38.482481    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 30 20:47:38.762893 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:38.757318    5071 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 20:47:38.762893 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:38.757381    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9" podUID="c80fd141-e109-4216-a3f1-39d326d72ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 20:47:39.029849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:39.029303    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 20:47:39.029849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:39.029361    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb" podUID="5cdad370-2ff5-435c-bbfe-2ba61a325d47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 20:47:39.190001 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:39.189911    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 20:47:39.190284 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:39.190256    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42" podUID="c4d634d9-39de-4614-9599-38adf86829e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 20:47:39.214996 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:39.214945    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:39.216091 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:39.215386    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 20:47:39.281106 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:39.281070    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 20:47:39.309554 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:39.309512    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk" podUID="6295120b-8bbc-4502-8429-4f04d7d5c5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 20:47:39.309775 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:39.283122    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 20:47:39.469903 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:39.455939    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:39.542516 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:39.541947    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:39.721901 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:39.718916    5071 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 20:47:39.721901 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:39.718979    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9" podUID="c80fd141-e109-4216-a3f1-39d326d72ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 20:47:40.075656 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:40.075611    5071 generic.go:356] "Generic (PLEG): container finished" podID="a3218c9e-b073-4137-9c34-26f2d0294ad0" containerID="5084c31899b706da6dea4857528ce6728bbd5b9030660b4652b22645fcb2419f" exitCode=0
Oct 30 20:47:40.076188 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:40.075908    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38cb58e4-bpnf4" event={"ID":"a3218c9e-b073-4137-9c34-26f2d0294ad0","Type":"ContainerDied","Data":"5084c31899b706da6dea4857528ce6728bbd5b9030660b4652b22645fcb2419f"}
Oct 30 20:47:40.088847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:40.085116    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:47:40.088847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:40.086595    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 20:47:40.088847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:40.086728    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb" podUID="5cdad370-2ff5-435c-bbfe-2ba61a325d47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 20:47:40.106514 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:40.102128    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:47:40.118846 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:40.114477    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c08ec9d-5mlgd" event={"ID":"4fde2656-e68e-4dfe-8bde-7138b60889e1","Type":"ContainerStarted","Data":"37b828b55918d43544b8bdb8137d2aceee80d1b85fa2aa8fd77b6a95a67374ed"}
Oct 30 20:47:40.173057 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:40.173018    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 20:47:40.211645 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:40.211603    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42" podUID="c4d634d9-39de-4614-9599-38adf86829e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 20:47:40.220365 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:40.220326    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 20:47:40.259894 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:40.233103    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 20:47:40.260232 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:40.260196    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk" podUID="6295120b-8bbc-4502-8429-4f04d7d5c5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 20:47:40.260429 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:40.237643    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:40.644964 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:40.627664    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:47:40.717467 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:40.717369    5071 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 20:47:40.717709 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:40.717676    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9" podUID="c80fd141-e109-4216-a3f1-39d326d72ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 20:47:40.944426 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:40.943795    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:47:40.991205 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:40.991120    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:47:41.028373 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:41.028336    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 20:47:41.028667 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:41.028639    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb" podUID="5cdad370-2ff5-435c-bbfe-2ba61a325d47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 20:47:41.152894 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:41.151150    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 20:47:41.152894 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:41.151212    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42" podUID="c4d634d9-39de-4614-9599-38adf86829e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 20:47:41.169858 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:41.166860    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e8761ca-97rdd"]
Oct 30 20:47:41.173118 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:41.173011    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8761ca-97rdd"
Oct 30 20:47:41.348928 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:41.342112    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8205cd06-0a45-4d84-99cb-0c0ca310a887-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e8761ca-97rdd\" (UID: \"8205cd06-0a45-4d84-99cb-0c0ca310a887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8761ca-97rdd"
Oct 30 20:47:41.353398 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:41.353362    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8205cd06-0a45-4d84-99cb-0c0ca310a887-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e8761ca-97rdd\" (UID: \"8205cd06-0a45-4d84-99cb-0c0ca310a887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8761ca-97rdd"
Oct 30 20:47:41.353615 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:41.353600    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmd9x\" (UniqueName: \"kubernetes.io/projected/8205cd06-0a45-4d84-99cb-0c0ca310a887-kube-api-access-hmd9x\") pod \"maas-api-revoke-sub-8e8761ca-97rdd\" (UID: \"8205cd06-0a45-4d84-99cb-0c0ca310a887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8761ca-97rdd"
Oct 30 20:47:41.382626 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:41.382558    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 20:47:41.382920 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:41.382885    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk" podUID="6295120b-8bbc-4502-8429-4f04d7d5c5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 20:47:41.510252 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:41.510220    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e8761ca-97rdd"]
Oct 30 20:47:41.510713 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:41.510633    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:41.538555 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:41.538493    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8205cd06-0a45-4d84-99cb-0c0ca310a887-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e8761ca-97rdd\" (UID: \"8205cd06-0a45-4d84-99cb-0c0ca310a887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8761ca-97rdd"
Oct 30 20:47:41.540404 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:41.540378    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmd9x\" (UniqueName: \"kubernetes.io/projected/8205cd06-0a45-4d84-99cb-0c0ca310a887-kube-api-access-hmd9x\") pod \"maas-api-revoke-sub-8e8761ca-97rdd\" (UID: \"8205cd06-0a45-4d84-99cb-0c0ca310a887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8761ca-97rdd"
Oct 30 20:47:41.542715 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:41.542692    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8205cd06-0a45-4d84-99cb-0c0ca310a887-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e8761ca-97rdd\" (UID: \"8205cd06-0a45-4d84-99cb-0c0ca310a887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8761ca-97rdd"
Oct 30 20:47:41.543596 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:41.539604    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8205cd06-0a45-4d84-99cb-0c0ca310a887-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e8761ca-97rdd\" (UID: \"8205cd06-0a45-4d84-99cb-0c0ca310a887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8761ca-97rdd"
Oct 30 20:47:41.543779 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:41.543039    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c08ec9d-5mlgd" event={"ID":"4fde2656-e68e-4dfe-8bde-7138b60889e1","Type":"ContainerStarted","Data":"31181f4bc6bded46f81703f090860136d71ecfdf46a2c4e94fcf738d02fa7525"}
Oct 30 20:47:41.546634 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:41.546613    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8205cd06-0a45-4d84-99cb-0c0ca310a887-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e8761ca-97rdd\" (UID: \"8205cd06-0a45-4d84-99cb-0c0ca310a887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8761ca-97rdd"
Oct 30 20:47:41.564972 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:41.564938    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:41.612979 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:41.612730    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmd9x\" (UniqueName: \"kubernetes.io/projected/8205cd06-0a45-4d84-99cb-0c0ca310a887-kube-api-access-hmd9x\") pod \"maas-api-revoke-sub-8e8761ca-97rdd\" (UID: \"8205cd06-0a45-4d84-99cb-0c0ca310a887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8761ca-97rdd"
Oct 30 20:47:41.697310 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:41.697237    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:47:41.733781 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:41.733570    5071 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 20:47:41.733781 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:41.733647    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9" podUID="c80fd141-e109-4216-a3f1-39d326d72ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 20:47:42.207895 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:42.207859    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 20:47:42.208690 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:42.208668    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 20:47:42.209066 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:42.209043    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk" podUID="6295120b-8bbc-4502-8429-4f04d7d5c5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 20:47:42.213695 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:42.208534    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42" podUID="c4d634d9-39de-4614-9599-38adf86829e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 20:47:42.511920 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:42.510994    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:42.591634 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:42.569196    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:42.662904 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:42.656128    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:47:42.663449 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:42.663430    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:47:42.677318 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:47:42.662546    5071 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fde2656_e68e_4dfe_8bde_7138b60889e1.slice/crio-conmon-31181f4bc6bded46f81703f090860136d71ecfdf46a2c4e94fcf738d02fa7525.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fde2656_e68e_4dfe_8bde_7138b60889e1.slice/crio-31181f4bc6bded46f81703f090860136d71ecfdf46a2c4e94fcf738d02fa7525.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:47:42.826302 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:42.805838    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:47:42.864913 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:42.863757    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8761ca-97rdd"
Oct 30 20:47:42.968727 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:42.965148    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:43.052367 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:43.051966    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:47:43.052367 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:43.052032    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb" podUID="5cdad370-2ff5-435c-bbfe-2ba61a325d47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:43.206931 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:43.149020    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 20:47:43.206931 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:43.149079    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42" podUID="c4d634d9-39de-4614-9599-38adf86829e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 20:47:43.206931 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:43.178064    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38cb58e4-bpnf4" event={"ID":"a3218c9e-b073-4137-9c34-26f2d0294ad0","Type":"ContainerDied","Data":"87ab5d94a223e130bc85bd3fcc8cbfe283cb110c04f7969b67d4739e292bd83e"}
Oct 30 20:47:43.206931 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:43.178127    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87ab5d94a223e130bc85bd3fcc8cbfe283cb110c04f7969b67d4739e292bd83e"
Oct 30 20:47:43.206931 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:43.204443    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 20:47:43.206931 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:43.204495    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk" podUID="6295120b-8bbc-4502-8429-4f04d7d5c5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 20:47:43.234973 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:43.234352    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:43.236020 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:43.235648    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38cb58e4-bpnf4"
Oct 30 20:47:43.317575 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:43.288117    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:47:43.346852 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:43.346416    5071 generic.go:356] "Generic (PLEG): container finished" podID="4fde2656-e68e-4dfe-8bde-7138b60889e1" containerID="31181f4bc6bded46f81703f090860136d71ecfdf46a2c4e94fcf738d02fa7525" exitCode=0
Oct 30 20:47:43.346852 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:43.346533    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c08ec9d-5mlgd" event={"ID":"4fde2656-e68e-4dfe-8bde-7138b60889e1","Type":"ContainerDied","Data":"31181f4bc6bded46f81703f090860136d71ecfdf46a2c4e94fcf738d02fa7525"}
Oct 30 20:47:43.382481 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:43.382445    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3218c9e-b073-4137-9c34-26f2d0294ad0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3218c9e-b073-4137-9c34-26f2d0294ad0-maas-api-openshift-ca\") pod \"a3218c9e-b073-4137-9c34-26f2d0294ad0\" (UID: \"a3218c9e-b073-4137-9c34-26f2d0294ad0\") "
Oct 30 20:47:43.386042 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:43.386012    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3218c9e-b073-4137-9c34-26f2d0294ad0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3218c9e-b073-4137-9c34-26f2d0294ad0-maas-api-xks-ca\") pod \"a3218c9e-b073-4137-9c34-26f2d0294ad0\" (UID: \"a3218c9e-b073-4137-9c34-26f2d0294ad0\") "
Oct 30 20:47:43.391152 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:43.391124    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3218c9e-b073-4137-9c34-26f2d0294ad0-kube-api-access-7mf45\" (UniqueName: \"kubernetes.io/projected/a3218c9e-b073-4137-9c34-26f2d0294ad0-kube-api-access-7mf45\") pod \"a3218c9e-b073-4137-9c34-26f2d0294ad0\" (UID: \"a3218c9e-b073-4137-9c34-26f2d0294ad0\") "
Oct 30 20:47:43.487235 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:43.431322    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3218c9e-b073-4137-9c34-26f2d0294ad0-maas-api-openshift-ca" pod "a3218c9e-b073-4137-9c34-26f2d0294ad0" (UID: "a3218c9e-b073-4137-9c34-26f2d0294ad0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:47:43.596887 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:43.582905    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3218c9e-b073-4137-9c34-26f2d0294ad0-maas-api-xks-ca" pod "a3218c9e-b073-4137-9c34-26f2d0294ad0" (UID: "a3218c9e-b073-4137-9c34-26f2d0294ad0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:47:43.622848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:43.612243    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3218c9e-b073-4137-9c34-26f2d0294ad0-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:43.623130 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:43.623103    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3218c9e-b073-4137-9c34-26f2d0294ad0-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:43.746070 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:43.745964    5071 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:47:43.746377 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:43.746325    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9" podUID="c80fd141-e109-4216-a3f1-39d326d72ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:43.788223 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:43.788188    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"
Oct 30 20:47:43.821851 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:43.806329    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"
Oct 30 20:47:43.965390 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:43.965313    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"
Oct 30 20:47:44.023084 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:44.022994    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1976c9ab-phg6d"]
Oct 30 20:47:44.242850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:44.237372    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 20:47:44.242850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:44.237451    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk" podUID="6295120b-8bbc-4502-8429-4f04d7d5c5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 20:47:44.321848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:44.309270    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:44.321848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:44.316266    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:44.352723 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:44.332501    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:44.410844 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:44.407560    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:44.410844 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:44.408029    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 20:47:44.410844 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:44.408097    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42" podUID="c4d634d9-39de-4614-9599-38adf86829e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 20:47:44.457160 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:44.457097    5071 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded"
Oct 30 20:47:44.718190 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:44.718135    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3218c9e-b073-4137-9c34-26f2d0294ad0-kube-api-access-7mf45" pod "a3218c9e-b073-4137-9c34-26f2d0294ad0" (UID: "a3218c9e-b073-4137-9c34-26f2d0294ad0"). InnerVolumeSpecName "kube-api-access-7mf45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:44.852850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:44.747106    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mf45\" (UniqueName: \"kubernetes.io/projected/a3218c9e-b073-4137-9c34-26f2d0294ad0-kube-api-access-7mf45\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:44.858557 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:44.801766    5071 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:47:44.863041 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:44.862976    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9" podUID="c80fd141-e109-4216-a3f1-39d326d72ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:44.877050 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:44.853197    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:44.908866 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:44.903997    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38cb58e4-bpnf4"
Oct 30 20:47:44.967849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:44.951781    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d445009-xqn4p"]
Oct 30 20:47:44.968127 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:44.968092    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1976c9ab-phg6d"]
Oct 30 20:47:44.968289 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:44.968273    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08207784-5kc2g"]
Oct 30 20:47:44.968513 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:44.968494    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:47:45.173902 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:45.170892    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 20:47:45.173902 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:45.170954    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42" podUID="c4d634d9-39de-4614-9599-38adf86829e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 20:47:45.264850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:45.230469    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 20:47:45.271210 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:45.271141    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk" podUID="6295120b-8bbc-4502-8429-4f04d7d5c5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 20:47:45.271512 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:45.268017    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d445009-xqn4p"]
Oct 30 20:47:45.374783 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:47:45.374725    5071 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3218c9e_b073_4137_9c34_26f2d0294ad0.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3218c9e_b073_4137_9c34_26f2d0294ad0.slice/crio-87ab5d94a223e130bc85bd3fcc8cbfe283cb110c04f7969b67d4739e292bd83e\": RecentStats: unable to find data in memory cache]"
Oct 30 20:47:45.423850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:45.399417    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 20:47:45.444520 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:45.415903    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:45.462076 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:45.417104    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:45.649848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:45.466279    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:47:45.674971 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:45.674940    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:47:45.682845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:45.678881    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08207784-5kc2g"]
Oct 30 20:47:45.882530 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:45.881956    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:45.980853 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:45.905839    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:47:46.077725 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:45.981177    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:46.078618 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:46.078574    5071 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:47:46.078964 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:46.078875    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9" podUID="c80fd141-e109-4216-a3f1-39d326d72ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:46.295901 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:46.269581    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 20:47:46.332893 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:46.325941    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk" podUID="6295120b-8bbc-4502-8429-4f04d7d5c5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 20:47:46.337533 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:46.337497    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa87e87b-e87a-4045-9266-7d4b720d14ea" path="/var/lib/kubelet/pods/aa87e87b-e87a-4045-9266-7d4b720d14ea/volumes"
Oct 30 20:47:46.338482 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:46.338457    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee132f80-47d9-49f2-ae94-b1d2b3c11c3a" path="/var/lib/kubelet/pods/ee132f80-47d9-49f2-ae94-b1d2b3c11c3a/volumes"
Oct 30 20:47:46.355508 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:46.355474    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f47fddaf-9a6c-4d08-beb4-3bef431ee07b" path="/var/lib/kubelet/pods/f47fddaf-9a6c-4d08-beb4-3bef431ee07b/volumes"
Oct 30 20:47:46.425177 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:46.425136    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:46.477480 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:46.466674    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:46.783947 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:46.779601    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:46.801518 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:46.793150    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"
Oct 30 20:47:46.910692 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:46.910638    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:46.997047 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:46.996990    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 20:47:46.997765 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:46.997693    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:47:47.116150 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:47.116092    5071 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": context deadline exceeded"
Oct 30 20:47:47.120983 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:47.120914    5071 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:47:47.124969 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:47.124911    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9" podUID="c80fd141-e109-4216-a3f1-39d326d72ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:47.159954 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:47.159925    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"
Oct 30 20:47:47.179793 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:47.179590    5071 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:47:47.196989 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:47.196931    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6" podUID="99069007-e0f8-4092-8fd6-f27b0a119720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:47.223986 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:47.223951    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:47:47.224356 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:47.224220    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42" podUID="c4d634d9-39de-4614-9599-38adf86829e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:47.258141 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:47.258037    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"
Oct 30 20:47:47.431429 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:47.431394    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 20:47:47.757472 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:47.721974    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk" podUID="6295120b-8bbc-4502-8429-4f04d7d5c5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 20:47:47.757472 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:47.603497    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:47.757472 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:47.542930    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 20:47:47.828167 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:47.828122    5071 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:47.851995 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:47.832120    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:47.912735 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:47.912691    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:47.962807 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:47:47.962616    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:47:47.987849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:47.971249    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c08ec9d-5mlgd"
Oct 30 20:47:48.061100 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:48.034762    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 20:47:48.267907 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:48.264476    5071 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:47:48.282881 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:48.268145    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9" podUID="c80fd141-e109-4216-a3f1-39d326d72ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:48.351306 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:48.350203    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:47:48.351306 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:48.350270    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42" podUID="c4d634d9-39de-4614-9599-38adf86829e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:48.555263 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:48.548292    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fde2656-e68e-4dfe-8bde-7138b60889e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fde2656-e68e-4dfe-8bde-7138b60889e1-maas-api-openshift-ca\") pod \"4fde2656-e68e-4dfe-8bde-7138b60889e1\" (UID: \"4fde2656-e68e-4dfe-8bde-7138b60889e1\") "
Oct 30 20:47:48.555263 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:48.548539    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fde2656-e68e-4dfe-8bde-7138b60889e1-kube-api-access-26zbt\" (UniqueName: \"kubernetes.io/projected/4fde2656-e68e-4dfe-8bde-7138b60889e1-kube-api-access-26zbt\") pod \"4fde2656-e68e-4dfe-8bde-7138b60889e1\" (UID: \"4fde2656-e68e-4dfe-8bde-7138b60889e1\") "
Oct 30 20:47:48.555263 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:48.548617    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fde2656-e68e-4dfe-8bde-7138b60889e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fde2656-e68e-4dfe-8bde-7138b60889e1-maas-api-xks-ca\") pod \"4fde2656-e68e-4dfe-8bde-7138b60889e1\" (UID: \"4fde2656-e68e-4dfe-8bde-7138b60889e1\") "
Oct 30 20:47:48.559936 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:48.557261    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fde2656-e68e-4dfe-8bde-7138b60889e1-maas-api-openshift-ca" pod "4fde2656-e68e-4dfe-8bde-7138b60889e1" (UID: "4fde2656-e68e-4dfe-8bde-7138b60889e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:47:48.772248 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:48.759267    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fde2656-e68e-4dfe-8bde-7138b60889e1-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:48.772248 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:48.760324    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:48.772248 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:48.760415    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 20:47:48.853964 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:48.831957    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fde2656-e68e-4dfe-8bde-7138b60889e1-maas-api-xks-ca" pod "4fde2656-e68e-4dfe-8bde-7138b60889e1" (UID: "4fde2656-e68e-4dfe-8bde-7138b60889e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:47:48.861796 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:48.861761    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fde2656-e68e-4dfe-8bde-7138b60889e1-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:48.872225 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:48.862088    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:48.885526 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:48.884062    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fde2656-e68e-4dfe-8bde-7138b60889e1-kube-api-access-26zbt" pod "4fde2656-e68e-4dfe-8bde-7138b60889e1" (UID: "4fde2656-e68e-4dfe-8bde-7138b60889e1"). InnerVolumeSpecName "kube-api-access-26zbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:49.003857 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:48.934964    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:49.005123 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:48.996812    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8761ca-97rdd" event={"ID":"8205cd06-0a45-4d84-99cb-0c0ca310a887","Type":"ContainerStarted","Data":"1f1a6494ba0e2e74ea6d3e782346dd8172cc3fcaddd5f8ef621f3baecd50e5f1"}
Oct 30 20:47:49.100843 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:49.069257    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:49.155917 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:49.102526    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26zbt\" (UniqueName: \"kubernetes.io/projected/4fde2656-e68e-4dfe-8bde-7138b60889e1-kube-api-access-26zbt\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:49.157567 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:49.101114    5071 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded"
Oct 30 20:47:49.247912 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:49.247255    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:47:49.247912 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:49.247323    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk" podUID="6295120b-8bbc-4502-8429-4f04d7d5c5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:49.357369 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:49.357329    5071 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:47:49.360855 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:49.357621    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9" podUID="c80fd141-e109-4216-a3f1-39d326d72ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:49.739084 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:49.683548    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"
Oct 30 20:47:49.946944 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:49.946897    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:49.977975 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:49.977923    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:50.155854 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:49.978299    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:47:50.185989 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:50.185914    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42" podUID="c4d634d9-39de-4614-9599-38adf86829e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:50.187676 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:49.981299    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 20:47:50.290849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:50.004698    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:50.290849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:50.090283    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e8761ca-97rdd"]
Oct 30 20:47:50.290849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:50.110182    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:50.317377 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:50.317337    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b43c84-7cc684fccf-k6nvt" podUID="b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Oct 30 20:47:50.479501 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:50.479409    5071 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:47:50.482972 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:50.482918    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9" podUID="c80fd141-e109-4216-a3f1-39d326d72ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:50.965850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:50.960556    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:51.025213 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:51.009583    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c08ec9d-5mlgd"
Oct 30 20:47:51.166907 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:51.150041    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:51.205888 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:51.200995    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:51.251087 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:51.242096    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:47:51.251087 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:51.242165    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42" podUID="c4d634d9-39de-4614-9599-38adf86829e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:51.276804 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:51.276759    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:51.312149 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:51.294120    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:51.546951 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:51.546563    5071 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:47:51.546951 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:51.546624    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9" podUID="c80fd141-e109-4216-a3f1-39d326d72ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:51.588614 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:47:51.588558    5071 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.129s"
Oct 30 20:47:51.589051 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:51.589034    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:47:51.589310 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:51.589297    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:47:51.589476 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:51.589457    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c08ec9d-5mlgd" event={"ID":"4fde2656-e68e-4dfe-8bde-7138b60889e1","Type":"ContainerDied","Data":"37b828b55918d43544b8bdb8137d2aceee80d1b85fa2aa8fd77b6a95a67374ed"}
Oct 30 20:47:51.589605 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:51.589592    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37b828b55918d43544b8bdb8137d2aceee80d1b85fa2aa8fd77b6a95a67374ed"
Oct 30 20:47:51.593478 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:51.590722    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8761ca-97rdd" podStartSLOduration=11.590705081 podStartE2EDuration="11.590705081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:51.356979554 +0000 UTC m=+1283.274001110" watchObservedRunningTime="2026-10-30 20:47:51.590705081 +0000 UTC m=+1283.507726618"
Oct 30 20:47:51.686600 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:51.686565    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:47:51.910902 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:51.878096    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"
Oct 30 20:47:51.910902 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:51.884960    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:47:52.012751 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:51.998099    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:47:52.013638 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:52.013616    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"
Oct 30 20:47:52.215227 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:52.215132    5071 generic.go:356] "Generic (PLEG): container finished" podID="8205cd06-0a45-4d84-99cb-0c0ca310a887" containerID="354e883b37c18432ea093778523ead22341b2289c37a8e9050d5d21a6800d856" exitCode=0
Oct 30 20:47:52.223703 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:52.223676    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8761ca-97rdd" event={"ID":"8205cd06-0a45-4d84-99cb-0c0ca310a887","Type":"ContainerDied","Data":"354e883b37c18432ea093778523ead22341b2289c37a8e9050d5d21a6800d856"}
Oct 30 20:47:52.269383 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:52.244512    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:47:52.269383 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:52.244568    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42" podUID="c4d634d9-39de-4614-9599-38adf86829e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:52.269383 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:52.244798    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:47:52.269383 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:52.244861    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk" podUID="6295120b-8bbc-4502-8429-4f04d7d5c5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:47:52.331096 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:52.328560    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"
Oct 30 20:47:52.334273 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:52.334231    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"
Oct 30 20:47:52.381802 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:52.381762    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:53.109642 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:53.109609    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-632ebc82-fggb4"]
Oct 30 20:47:53.110415 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:53.110396    5071 cpu_manager.go:414] "containerMap: removing container" podUID="4fde2656-e68e-4dfe-8bde-7138b60889e1" containerName="revoke-keys"
Oct 30 20:47:53.110473 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:53.110417    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fde2656-e68e-4dfe-8bde-7138b60889e1" containerName="revoke-keys"
Oct 30 20:47:53.110473 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:53.110452    5071 cpu_manager.go:414] "containerMap: removing container" podUID="a3218c9e-b073-4137-9c34-26f2d0294ad0" containerName="revoke-keys"
Oct 30 20:47:53.110473 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:53.110458    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3218c9e-b073-4137-9c34-26f2d0294ad0" containerName="revoke-keys"
Oct 30 20:47:53.110578 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:53.110564    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fde2656-e68e-4dfe-8bde-7138b60889e1" containerName="revoke-keys"
Oct 30 20:47:53.110608 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:53.110578    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3218c9e-b073-4137-9c34-26f2d0294ad0" containerName="revoke-keys"
Oct 30 20:47:53.114416 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:53.114399    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-632ebc82-fggb4"
Oct 30 20:47:53.175845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:53.175661    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-632ebc82-fggb4"]
Oct 30 20:47:53.251202 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:53.251160    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xm4h\" (UniqueName: \"kubernetes.io/projected/5c164e31-eaeb-49fa-bcae-17934ed9febc-kube-api-access-9xm4h\") pod \"maas-api-revoke-sub-632ebc82-fggb4\" (UID: \"5c164e31-eaeb-49fa-bcae-17934ed9febc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-632ebc82-fggb4"
Oct 30 20:47:53.251415 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:53.251388    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c164e31-eaeb-49fa-bcae-17934ed9febc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-632ebc82-fggb4\" (UID: \"5c164e31-eaeb-49fa-bcae-17934ed9febc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-632ebc82-fggb4"
Oct 30 20:47:53.251494 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:53.251453    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c164e31-eaeb-49fa-bcae-17934ed9febc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-632ebc82-fggb4\" (UID: \"5c164e31-eaeb-49fa-bcae-17934ed9febc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-632ebc82-fggb4"
Oct 30 20:47:53.352683 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:53.352636    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xm4h\" (UniqueName: \"kubernetes.io/projected/5c164e31-eaeb-49fa-bcae-17934ed9febc-kube-api-access-9xm4h\") pod \"maas-api-revoke-sub-632ebc82-fggb4\" (UID: \"5c164e31-eaeb-49fa-bcae-17934ed9febc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-632ebc82-fggb4"
Oct 30 20:47:53.352924 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:53.352903    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c164e31-eaeb-49fa-bcae-17934ed9febc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-632ebc82-fggb4\" (UID: \"5c164e31-eaeb-49fa-bcae-17934ed9febc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-632ebc82-fggb4"
Oct 30 20:47:53.353010 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:53.352952    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c164e31-eaeb-49fa-bcae-17934ed9febc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-632ebc82-fggb4\" (UID: \"5c164e31-eaeb-49fa-bcae-17934ed9febc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-632ebc82-fggb4"
Oct 30 20:47:53.354476 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:53.354447    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c164e31-eaeb-49fa-bcae-17934ed9febc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-632ebc82-fggb4\" (UID: \"5c164e31-eaeb-49fa-bcae-17934ed9febc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-632ebc82-fggb4"
Oct 30 20:47:53.355899 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:53.355877    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c164e31-eaeb-49fa-bcae-17934ed9febc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-632ebc82-fggb4\" (UID: \"5c164e31-eaeb-49fa-bcae-17934ed9febc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-632ebc82-fggb4"
Oct 30 20:47:53.411775 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:53.411743    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xm4h\" (UniqueName: \"kubernetes.io/projected/5c164e31-eaeb-49fa-bcae-17934ed9febc-kube-api-access-9xm4h\") pod \"maas-api-revoke-sub-632ebc82-fggb4\" (UID: \"5c164e31-eaeb-49fa-bcae-17934ed9febc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-632ebc82-fggb4"
Oct 30 20:47:53.428266 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:53.428240    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-632ebc82-fggb4"
Oct 30 20:47:53.434007 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:53.433984    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8761ca-97rdd"
Oct 30 20:47:53.456550 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:53.456522    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8205cd06-0a45-4d84-99cb-0c0ca310a887-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8205cd06-0a45-4d84-99cb-0c0ca310a887-maas-api-openshift-ca\") pod \"8205cd06-0a45-4d84-99cb-0c0ca310a887\" (UID: \"8205cd06-0a45-4d84-99cb-0c0ca310a887\") "
Oct 30 20:47:53.457262 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:53.456995    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8205cd06-0a45-4d84-99cb-0c0ca310a887-maas-api-openshift-ca" pod "8205cd06-0a45-4d84-99cb-0c0ca310a887" (UID: "8205cd06-0a45-4d84-99cb-0c0ca310a887"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:47:53.457262 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:53.457151    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8205cd06-0a45-4d84-99cb-0c0ca310a887-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:53.569216 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:53.565974    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8205cd06-0a45-4d84-99cb-0c0ca310a887-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8205cd06-0a45-4d84-99cb-0c0ca310a887-maas-api-xks-ca\") pod \"8205cd06-0a45-4d84-99cb-0c0ca310a887\" (UID: \"8205cd06-0a45-4d84-99cb-0c0ca310a887\") "
Oct 30 20:47:53.569216 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:53.566094    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8205cd06-0a45-4d84-99cb-0c0ca310a887-kube-api-access-hmd9x\" (UniqueName: \"kubernetes.io/projected/8205cd06-0a45-4d84-99cb-0c0ca310a887-kube-api-access-hmd9x\") pod \"8205cd06-0a45-4d84-99cb-0c0ca310a887\" (UID: \"8205cd06-0a45-4d84-99cb-0c0ca310a887\") "
Oct 30 20:47:53.578184 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:53.578130    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8205cd06-0a45-4d84-99cb-0c0ca310a887-maas-api-xks-ca" pod "8205cd06-0a45-4d84-99cb-0c0ca310a887" (UID: "8205cd06-0a45-4d84-99cb-0c0ca310a887"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:47:53.579170 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:53.579144    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8205cd06-0a45-4d84-99cb-0c0ca310a887-kube-api-access-hmd9x" pod "8205cd06-0a45-4d84-99cb-0c0ca310a887" (UID: "8205cd06-0a45-4d84-99cb-0c0ca310a887"). InnerVolumeSpecName "kube-api-access-hmd9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:53.668400 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:53.668316    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmd9x\" (UniqueName: \"kubernetes.io/projected/8205cd06-0a45-4d84-99cb-0c0ca310a887-kube-api-access-hmd9x\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:53.668400 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:53.668345    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8205cd06-0a45-4d84-99cb-0c0ca310a887-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:53.815701 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:47:53.815670    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c164e31_eaeb_49fa_bcae_17934ed9febc.slice/crio-9a9e91c64e40165a83dc28dccfcb9c9fb05ae2c7457f88e2163f8676cfd4c17e WatchSource:0}: Error finding container 9a9e91c64e40165a83dc28dccfcb9c9fb05ae2c7457f88e2163f8676cfd4c17e: Status 404 returned error can't find the container with id 9a9e91c64e40165a83dc28dccfcb9c9fb05ae2c7457f88e2163f8676cfd4c17e
Oct 30 20:47:53.816809 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:53.816781    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-632ebc82-fggb4"]
Oct 30 20:47:54.233894 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:54.233855    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-632ebc82-fggb4" event={"ID":"5c164e31-eaeb-49fa-bcae-17934ed9febc","Type":"ContainerStarted","Data":"9a9e91c64e40165a83dc28dccfcb9c9fb05ae2c7457f88e2163f8676cfd4c17e"}
Oct 30 20:47:54.235797 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:54.235770    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8761ca-97rdd" event={"ID":"8205cd06-0a45-4d84-99cb-0c0ca310a887","Type":"ContainerDied","Data":"1f1a6494ba0e2e74ea6d3e782346dd8172cc3fcaddd5f8ef621f3baecd50e5f1"}
Oct 30 20:47:54.235923 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:54.235802    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8761ca-97rdd"
Oct 30 20:47:54.235923 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:54.235816    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f1a6494ba0e2e74ea6d3e782346dd8172cc3fcaddd5f8ef621f3baecd50e5f1"
Oct 30 20:47:55.249419 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:55.249386    5071 generic.go:356] "Generic (PLEG): container finished" podID="5c164e31-eaeb-49fa-bcae-17934ed9febc" containerID="93e4c63fd0cc4b8d0a55600b38159631c4bc474ba48db35f878ba6c2f8a357cc" exitCode=0
Oct 30 20:47:55.249883 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:55.249471    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-632ebc82-fggb4" event={"ID":"5c164e31-eaeb-49fa-bcae-17934ed9febc","Type":"ContainerDied","Data":"93e4c63fd0cc4b8d0a55600b38159631c4bc474ba48db35f878ba6c2f8a357cc"}
Oct 30 20:47:56.457782 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:56.457751    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-632ebc82-fggb4"
Oct 30 20:47:56.513551 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:56.513473    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c164e31-eaeb-49fa-bcae-17934ed9febc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c164e31-eaeb-49fa-bcae-17934ed9febc-maas-api-openshift-ca\") pod \"5c164e31-eaeb-49fa-bcae-17934ed9febc\" (UID: \"5c164e31-eaeb-49fa-bcae-17934ed9febc\") "
Oct 30 20:47:56.513723 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:56.513600    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c164e31-eaeb-49fa-bcae-17934ed9febc-kube-api-access-9xm4h\" (UniqueName: \"kubernetes.io/projected/5c164e31-eaeb-49fa-bcae-17934ed9febc-kube-api-access-9xm4h\") pod \"5c164e31-eaeb-49fa-bcae-17934ed9febc\" (UID: \"5c164e31-eaeb-49fa-bcae-17934ed9febc\") "
Oct 30 20:47:56.513723 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:56.513636    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c164e31-eaeb-49fa-bcae-17934ed9febc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c164e31-eaeb-49fa-bcae-17934ed9febc-maas-api-xks-ca\") pod \"5c164e31-eaeb-49fa-bcae-17934ed9febc\" (UID: \"5c164e31-eaeb-49fa-bcae-17934ed9febc\") "
Oct 30 20:47:56.513932 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:56.513909    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c164e31-eaeb-49fa-bcae-17934ed9febc-maas-api-openshift-ca" pod "5c164e31-eaeb-49fa-bcae-17934ed9febc" (UID: "5c164e31-eaeb-49fa-bcae-17934ed9febc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:47:56.514267 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:56.514242    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c164e31-eaeb-49fa-bcae-17934ed9febc-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:56.515844 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:56.515804    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c164e31-eaeb-49fa-bcae-17934ed9febc-maas-api-xks-ca" pod "5c164e31-eaeb-49fa-bcae-17934ed9febc" (UID: "5c164e31-eaeb-49fa-bcae-17934ed9febc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:47:56.515934 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:56.515885    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c164e31-eaeb-49fa-bcae-17934ed9febc-kube-api-access-9xm4h" pod "5c164e31-eaeb-49fa-bcae-17934ed9febc" (UID: "5c164e31-eaeb-49fa-bcae-17934ed9febc"). InnerVolumeSpecName "kube-api-access-9xm4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:56.616653 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:56.616597    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xm4h\" (UniqueName: \"kubernetes.io/projected/5c164e31-eaeb-49fa-bcae-17934ed9febc-kube-api-access-9xm4h\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:56.616863 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:56.616672    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c164e31-eaeb-49fa-bcae-17934ed9febc-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:57.267675 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:57.267649    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-632ebc82-fggb4"
Oct 30 20:47:57.267864 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:57.267645    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-632ebc82-fggb4" event={"ID":"5c164e31-eaeb-49fa-bcae-17934ed9febc","Type":"ContainerDied","Data":"9a9e91c64e40165a83dc28dccfcb9c9fb05ae2c7457f88e2163f8676cfd4c17e"}
Oct 30 20:47:57.267864 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:57.267779    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a9e91c64e40165a83dc28dccfcb9c9fb05ae2c7457f88e2163f8676cfd4c17e"
Oct 30 20:47:57.413992 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:57.410530    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:47:59.474136 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:59.474101    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3114635a-nvgjq"]
Oct 30 20:47:59.474778 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:59.474764    5071 cpu_manager.go:414] "containerMap: removing container" podUID="8205cd06-0a45-4d84-99cb-0c0ca310a887" containerName="revoke-keys"
Oct 30 20:47:59.474872 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:59.474779    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8205cd06-0a45-4d84-99cb-0c0ca310a887" containerName="revoke-keys"
Oct 30 20:47:59.474872 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:59.474793    5071 cpu_manager.go:414] "containerMap: removing container" podUID="5c164e31-eaeb-49fa-bcae-17934ed9febc" containerName="revoke-keys"
Oct 30 20:47:59.474872 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:59.474798    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c164e31-eaeb-49fa-bcae-17934ed9febc" containerName="revoke-keys"
Oct 30 20:47:59.474957 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:59.474932    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="8205cd06-0a45-4d84-99cb-0c0ca310a887" containerName="revoke-keys"
Oct 30 20:47:59.474957 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:59.474940    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c164e31-eaeb-49fa-bcae-17934ed9febc" containerName="revoke-keys"
Oct 30 20:47:59.479641 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:59.479624    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3114635a-nvgjq"
Oct 30 20:47:59.537753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:59.537720    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3114635a-nvgjq"]
Oct 30 20:47:59.571567 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:59.571530    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb2c533f-9768-436d-bddc-4de7e46885ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3114635a-nvgjq\" (UID: \"cb2c533f-9768-436d-bddc-4de7e46885ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3114635a-nvgjq"
Oct 30 20:47:59.571567 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:59.571572    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7cb9\" (UniqueName: \"kubernetes.io/projected/cb2c533f-9768-436d-bddc-4de7e46885ec-kube-api-access-s7cb9\") pod \"maas-api-revoke-sub-3114635a-nvgjq\" (UID: \"cb2c533f-9768-436d-bddc-4de7e46885ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3114635a-nvgjq"
Oct 30 20:47:59.571849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:59.571673    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb2c533f-9768-436d-bddc-4de7e46885ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3114635a-nvgjq\" (UID: \"cb2c533f-9768-436d-bddc-4de7e46885ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3114635a-nvgjq"
Oct 30 20:47:59.581467 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:59.581442    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:47:59.581943 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:59.581916    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 20:47:59.663367 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:59.663340    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddcb2e1b-qbrbt"]
Oct 30 20:47:59.668146 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:59.668127    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddcb2e1b-qbrbt"
Oct 30 20:47:59.672766 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:59.672742    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb2c533f-9768-436d-bddc-4de7e46885ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3114635a-nvgjq\" (UID: \"cb2c533f-9768-436d-bddc-4de7e46885ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3114635a-nvgjq"
Oct 30 20:47:59.672895 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:59.672779    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7cb9\" (UniqueName: \"kubernetes.io/projected/cb2c533f-9768-436d-bddc-4de7e46885ec-kube-api-access-s7cb9\") pod \"maas-api-revoke-sub-3114635a-nvgjq\" (UID: \"cb2c533f-9768-436d-bddc-4de7e46885ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3114635a-nvgjq"
Oct 30 20:47:59.672895 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:59.672806    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb2c533f-9768-436d-bddc-4de7e46885ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3114635a-nvgjq\" (UID: \"cb2c533f-9768-436d-bddc-4de7e46885ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3114635a-nvgjq"
Oct 30 20:47:59.673631 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:59.673601    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb2c533f-9768-436d-bddc-4de7e46885ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3114635a-nvgjq\" (UID: \"cb2c533f-9768-436d-bddc-4de7e46885ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3114635a-nvgjq"
Oct 30 20:47:59.675366 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:59.675345    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb2c533f-9768-436d-bddc-4de7e46885ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3114635a-nvgjq\" (UID: \"cb2c533f-9768-436d-bddc-4de7e46885ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3114635a-nvgjq"
Oct 30 20:47:59.730514 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:59.730454    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddcb2e1b-qbrbt"]
Oct 30 20:47:59.732236 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:59.732211    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7cb9\" (UniqueName: \"kubernetes.io/projected/cb2c533f-9768-436d-bddc-4de7e46885ec-kube-api-access-s7cb9\") pod \"maas-api-revoke-sub-3114635a-nvgjq\" (UID: \"cb2c533f-9768-436d-bddc-4de7e46885ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3114635a-nvgjq"
Oct 30 20:47:59.774173 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:59.774133    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a90f40d0-8846-469d-b3e9-c9158d667dfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddcb2e1b-qbrbt\" (UID: \"a90f40d0-8846-469d-b3e9-c9158d667dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddcb2e1b-qbrbt"
Oct 30 20:47:59.774336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:59.774248    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8m4l\" (UniqueName: \"kubernetes.io/projected/a90f40d0-8846-469d-b3e9-c9158d667dfa-kube-api-access-c8m4l\") pod \"maas-api-revoke-sub-ddcb2e1b-qbrbt\" (UID: \"a90f40d0-8846-469d-b3e9-c9158d667dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddcb2e1b-qbrbt"
Oct 30 20:47:59.774336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:59.774303    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a90f40d0-8846-469d-b3e9-c9158d667dfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddcb2e1b-qbrbt\" (UID: \"a90f40d0-8846-469d-b3e9-c9158d667dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddcb2e1b-qbrbt"
Oct 30 20:47:59.794043 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:59.794014    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3114635a-nvgjq"
Oct 30 20:47:59.880115 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:59.880080    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a90f40d0-8846-469d-b3e9-c9158d667dfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddcb2e1b-qbrbt\" (UID: \"a90f40d0-8846-469d-b3e9-c9158d667dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddcb2e1b-qbrbt"
Oct 30 20:47:59.880258 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:59.880176    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8m4l\" (UniqueName: \"kubernetes.io/projected/a90f40d0-8846-469d-b3e9-c9158d667dfa-kube-api-access-c8m4l\") pod \"maas-api-revoke-sub-ddcb2e1b-qbrbt\" (UID: \"a90f40d0-8846-469d-b3e9-c9158d667dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddcb2e1b-qbrbt"
Oct 30 20:47:59.880258 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:59.880223    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a90f40d0-8846-469d-b3e9-c9158d667dfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddcb2e1b-qbrbt\" (UID: \"a90f40d0-8846-469d-b3e9-c9158d667dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddcb2e1b-qbrbt"
Oct 30 20:47:59.881234 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:59.881204    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a90f40d0-8846-469d-b3e9-c9158d667dfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddcb2e1b-qbrbt\" (UID: \"a90f40d0-8846-469d-b3e9-c9158d667dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddcb2e1b-qbrbt"
Oct 30 20:47:59.883620 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:59.883597    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a90f40d0-8846-469d-b3e9-c9158d667dfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddcb2e1b-qbrbt\" (UID: \"a90f40d0-8846-469d-b3e9-c9158d667dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddcb2e1b-qbrbt"
Oct 30 20:47:59.941584 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:59.941551    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8m4l\" (UniqueName: \"kubernetes.io/projected/a90f40d0-8846-469d-b3e9-c9158d667dfa-kube-api-access-c8m4l\") pod \"maas-api-revoke-sub-ddcb2e1b-qbrbt\" (UID: \"a90f40d0-8846-469d-b3e9-c9158d667dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddcb2e1b-qbrbt"
Oct 30 20:47:59.998434 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:47:59.998347    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddcb2e1b-qbrbt"
Oct 30 20:48:00.188721 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:48:00.188684    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb2c533f_9768_436d_bddc_4de7e46885ec.slice/crio-3eb200531e64267e658dd8756dcfc7b3e3b1c7c698cb0951f14773b76fdd6272 WatchSource:0}: Error finding container 3eb200531e64267e658dd8756dcfc7b3e3b1c7c698cb0951f14773b76fdd6272: Status 404 returned error can't find the container with id 3eb200531e64267e658dd8756dcfc7b3e3b1c7c698cb0951f14773b76fdd6272
Oct 30 20:48:00.189438 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.189404    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3114635a-nvgjq"]
Oct 30 20:48:00.293648 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.293612    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3114635a-nvgjq" event={"ID":"cb2c533f-9768-436d-bddc-4de7e46885ec","Type":"ContainerStarted","Data":"3eb200531e64267e658dd8756dcfc7b3e3b1c7c698cb0951f14773b76fdd6272"}
Oct 30 20:48:00.302166 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.302129    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"]
Oct 30 20:48:00.307736 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.307713    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"
Oct 30 20:48:00.333491 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.333466    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"]
Oct 30 20:48:00.338139 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.338121    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"
Oct 30 20:48:00.369303 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.369269    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"]
Oct 30 20:48:00.387449 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.387428    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddcb2e1b-qbrbt"]
Oct 30 20:48:00.388122 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.388099    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4349a44e-61fd-4a25-8548-96b11c3cf4cc-workload-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv\" (UID: \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"
Oct 30 20:48:00.388205 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.388161    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4349a44e-61fd-4a25-8548-96b11c3cf4cc-istio-podinfo\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv\" (UID: \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"
Oct 30 20:48:00.388278 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.388263    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4349a44e-61fd-4a25-8548-96b11c3cf4cc-credential-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv\" (UID: \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"
Oct 30 20:48:00.388317 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.388302    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55a2c070-6473-481d-a202-f9fcd1231856-istio-data\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5\" (UID: \"55a2c070-6473-481d-a202-f9fcd1231856\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"
Oct 30 20:48:00.388368 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.388324    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55a2c070-6473-481d-a202-f9fcd1231856-istio-envoy\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5\" (UID: \"55a2c070-6473-481d-a202-f9fcd1231856\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"
Oct 30 20:48:00.388402 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.388389    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd27m\" (UniqueName: \"kubernetes.io/projected/4349a44e-61fd-4a25-8548-96b11c3cf4cc-kube-api-access-fd27m\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv\" (UID: \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"
Oct 30 20:48:00.388492 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.388478    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55a2c070-6473-481d-a202-f9fcd1231856-istio-podinfo\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5\" (UID: \"55a2c070-6473-481d-a202-f9fcd1231856\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"
Oct 30 20:48:00.388531 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.388512    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4v6n\" (UniqueName: \"kubernetes.io/projected/55a2c070-6473-481d-a202-f9fcd1231856-kube-api-access-k4v6n\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5\" (UID: \"55a2c070-6473-481d-a202-f9fcd1231856\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"
Oct 30 20:48:00.388607 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.388589    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4349a44e-61fd-4a25-8548-96b11c3cf4cc-istio-data\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv\" (UID: \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"
Oct 30 20:48:00.388653 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.388638    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4349a44e-61fd-4a25-8548-96b11c3cf4cc-workload-certs\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv\" (UID: \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"
Oct 30 20:48:00.388683 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.388668    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55a2c070-6473-481d-a202-f9fcd1231856-workload-certs\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5\" (UID: \"55a2c070-6473-481d-a202-f9fcd1231856\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"
Oct 30 20:48:00.388713 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.388688    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55a2c070-6473-481d-a202-f9fcd1231856-istiod-ca-cert\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5\" (UID: \"55a2c070-6473-481d-a202-f9fcd1231856\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"
Oct 30 20:48:00.388761 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.388748    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a2c070-6473-481d-a202-f9fcd1231856-credential-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5\" (UID: \"55a2c070-6473-481d-a202-f9fcd1231856\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"
Oct 30 20:48:00.388796 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.388770    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4349a44e-61fd-4a25-8548-96b11c3cf4cc-istio-token\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv\" (UID: \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"
Oct 30 20:48:00.388867 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.388834    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55a2c070-6473-481d-a202-f9fcd1231856-istio-token\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5\" (UID: \"55a2c070-6473-481d-a202-f9fcd1231856\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"
Oct 30 20:48:00.388908 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.388870    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a2c070-6473-481d-a202-f9fcd1231856-workload-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5\" (UID: \"55a2c070-6473-481d-a202-f9fcd1231856\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"
Oct 30 20:48:00.388908 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.388888    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4349a44e-61fd-4a25-8548-96b11c3cf4cc-istiod-ca-cert\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv\" (UID: \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"
Oct 30 20:48:00.388978 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.388965    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4349a44e-61fd-4a25-8548-96b11c3cf4cc-istio-envoy\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv\" (UID: \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"
Oct 30 20:48:00.462113 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.462072    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"]
Oct 30 20:48:00.489812 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.489783    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4349a44e-61fd-4a25-8548-96b11c3cf4cc-workload-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv\" (UID: \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"
Oct 30 20:48:00.490234 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.489869    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4349a44e-61fd-4a25-8548-96b11c3cf4cc-istio-podinfo\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv\" (UID: \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"
Oct 30 20:48:00.490234 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.489959    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4349a44e-61fd-4a25-8548-96b11c3cf4cc-credential-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv\" (UID: \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"
Oct 30 20:48:00.490234 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.490007    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55a2c070-6473-481d-a202-f9fcd1231856-istio-data\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5\" (UID: \"55a2c070-6473-481d-a202-f9fcd1231856\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"
Oct 30 20:48:00.490234 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.490038    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55a2c070-6473-481d-a202-f9fcd1231856-istio-envoy\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5\" (UID: \"55a2c070-6473-481d-a202-f9fcd1231856\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"
Oct 30 20:48:00.490234 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.490063    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd27m\" (UniqueName: \"kubernetes.io/projected/4349a44e-61fd-4a25-8548-96b11c3cf4cc-kube-api-access-fd27m\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv\" (UID: \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"
Oct 30 20:48:00.490234 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.490104    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55a2c070-6473-481d-a202-f9fcd1231856-istio-podinfo\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5\" (UID: \"55a2c070-6473-481d-a202-f9fcd1231856\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"
Oct 30 20:48:00.490234 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.490150    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4v6n\" (UniqueName: \"kubernetes.io/projected/55a2c070-6473-481d-a202-f9fcd1231856-kube-api-access-k4v6n\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5\" (UID: \"55a2c070-6473-481d-a202-f9fcd1231856\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"
Oct 30 20:48:00.490234 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.490178    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4349a44e-61fd-4a25-8548-96b11c3cf4cc-istio-data\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv\" (UID: \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"
Oct 30 20:48:00.490234 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.490210    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4349a44e-61fd-4a25-8548-96b11c3cf4cc-workload-certs\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv\" (UID: \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"
Oct 30 20:48:00.490234 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.490228    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4349a44e-61fd-4a25-8548-96b11c3cf4cc-workload-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv\" (UID: \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"
Oct 30 20:48:00.490740 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.490240    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55a2c070-6473-481d-a202-f9fcd1231856-workload-certs\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5\" (UID: \"55a2c070-6473-481d-a202-f9fcd1231856\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"
Oct 30 20:48:00.490740 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.490292    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55a2c070-6473-481d-a202-f9fcd1231856-istiod-ca-cert\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5\" (UID: \"55a2c070-6473-481d-a202-f9fcd1231856\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"
Oct 30 20:48:00.490740 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.490366    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a2c070-6473-481d-a202-f9fcd1231856-credential-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5\" (UID: \"55a2c070-6473-481d-a202-f9fcd1231856\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"
Oct 30 20:48:00.490740 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.490432    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4349a44e-61fd-4a25-8548-96b11c3cf4cc-istio-token\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv\" (UID: \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"
Oct 30 20:48:00.490740 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.490501    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55a2c070-6473-481d-a202-f9fcd1231856-istio-token\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5\" (UID: \"55a2c070-6473-481d-a202-f9fcd1231856\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"
Oct 30 20:48:00.490740 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.490566    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a2c070-6473-481d-a202-f9fcd1231856-workload-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5\" (UID: \"55a2c070-6473-481d-a202-f9fcd1231856\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"
Oct 30 20:48:00.490740 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.490593    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55a2c070-6473-481d-a202-f9fcd1231856-workload-certs\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5\" (UID: \"55a2c070-6473-481d-a202-f9fcd1231856\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"
Oct 30 20:48:00.490740 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.490597    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4349a44e-61fd-4a25-8548-96b11c3cf4cc-istiod-ca-cert\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv\" (UID: \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"
Oct 30 20:48:00.490740 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.490689    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4349a44e-61fd-4a25-8548-96b11c3cf4cc-istio-envoy\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv\" (UID: \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"
Oct 30 20:48:00.491210 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.491008    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55a2c070-6473-481d-a202-f9fcd1231856-istiod-ca-cert\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5\" (UID: \"55a2c070-6473-481d-a202-f9fcd1231856\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"
Oct 30 20:48:00.491430 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.491376    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a2c070-6473-481d-a202-f9fcd1231856-credential-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5\" (UID: \"55a2c070-6473-481d-a202-f9fcd1231856\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"
Oct 30 20:48:00.492242 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.491938    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4349a44e-61fd-4a25-8548-96b11c3cf4cc-credential-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv\" (UID: \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"
Oct 30 20:48:00.492242 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.492116    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a2c070-6473-481d-a202-f9fcd1231856-workload-socket\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5\" (UID: \"55a2c070-6473-481d-a202-f9fcd1231856\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"
Oct 30 20:48:00.492242 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.492138    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4349a44e-61fd-4a25-8548-96b11c3cf4cc-istiod-ca-cert\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv\" (UID: \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"
Oct 30 20:48:00.492242 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.492206    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4349a44e-61fd-4a25-8548-96b11c3cf4cc-istio-data\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv\" (UID: \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"
Oct 30 20:48:00.492514 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.492479    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4349a44e-61fd-4a25-8548-96b11c3cf4cc-workload-certs\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv\" (UID: \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"
Oct 30 20:48:00.492589 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.492499    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55a2c070-6473-481d-a202-f9fcd1231856-istio-data\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5\" (UID: \"55a2c070-6473-481d-a202-f9fcd1231856\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"
Oct 30 20:48:00.493131 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.493107    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4349a44e-61fd-4a25-8548-96b11c3cf4cc-istio-envoy\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv\" (UID: \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"
Oct 30 20:48:00.493605 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.493582    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4349a44e-61fd-4a25-8548-96b11c3cf4cc-istio-podinfo\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv\" (UID: \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"
Oct 30 20:48:00.494977 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.494953    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55a2c070-6473-481d-a202-f9fcd1231856-istio-envoy\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5\" (UID: \"55a2c070-6473-481d-a202-f9fcd1231856\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"
Oct 30 20:48:00.495221 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.495201    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55a2c070-6473-481d-a202-f9fcd1231856-istio-podinfo\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5\" (UID: \"55a2c070-6473-481d-a202-f9fcd1231856\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"
Oct 30 20:48:00.555645 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.555566    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd27m\" (UniqueName: \"kubernetes.io/projected/4349a44e-61fd-4a25-8548-96b11c3cf4cc-kube-api-access-fd27m\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv\" (UID: \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"
Oct 30 20:48:00.558227 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.558197    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4349a44e-61fd-4a25-8548-96b11c3cf4cc-istio-token\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv\" (UID: \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"
Oct 30 20:48:00.559033 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.559001    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55a2c070-6473-481d-a202-f9fcd1231856-istio-token\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5\" (UID: \"55a2c070-6473-481d-a202-f9fcd1231856\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"
Oct 30 20:48:00.559134 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.559077    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4v6n\" (UniqueName: \"kubernetes.io/projected/55a2c070-6473-481d-a202-f9fcd1231856-kube-api-access-k4v6n\") pod \"e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5\" (UID: \"55a2c070-6473-481d-a202-f9fcd1231856\") " pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"
Oct 30 20:48:00.622766 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.622729    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"
Oct 30 20:48:00.653120 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:00.653023    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"
Oct 30 20:48:01.073876 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:48:01.073845    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55a2c070_6473_481d_a202_f9fcd1231856.slice/crio-0b9ebdc57202f3a59a73a9bc8456cf75b866714f54e9e3420b1f59e637fdb24b WatchSource:0}: Error finding container 0b9ebdc57202f3a59a73a9bc8456cf75b866714f54e9e3420b1f59e637fdb24b: Status 404 returned error can't find the container with id 0b9ebdc57202f3a59a73a9bc8456cf75b866714f54e9e3420b1f59e637fdb24b
Oct 30 20:48:01.074180 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:01.074154    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"]
Oct 30 20:48:01.076902 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:01.076864    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:48:01.076998 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:01.076944    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:48:01.076998 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:01.076987    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:48:01.128802 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:01.127295    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"]
Oct 30 20:48:01.128802 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:48:01.127662    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4349a44e_61fd_4a25_8548_96b11c3cf4cc.slice/crio-0132177a676b6cf677275ff3c267cc1e5b5d246a832fa0bf641535de02067048 WatchSource:0}: Error finding container 0132177a676b6cf677275ff3c267cc1e5b5d246a832fa0bf641535de02067048: Status 404 returned error can't find the container with id 0132177a676b6cf677275ff3c267cc1e5b5d246a832fa0bf641535de02067048
Oct 30 20:48:01.131383 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:01.131346    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:48:01.131495 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:01.131416    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:48:01.131495 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:01.131451    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:48:01.307460 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:01.307421    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3114635a-nvgjq" event={"ID":"cb2c533f-9768-436d-bddc-4de7e46885ec","Type":"ContainerStarted","Data":"b7f3f344358e8524aeeac5a775d512042c711dcd3691fb4fc4a5174ca24cc58f"}
Oct 30 20:48:01.309774 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:01.309748    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddcb2e1b-qbrbt" event={"ID":"a90f40d0-8846-469d-b3e9-c9158d667dfa","Type":"ContainerStarted","Data":"f37ccb8ca15b318706c0c7fd8e5bd6142730a5c4712156d5fa0a73f0ba1ff44f"}
Oct 30 20:48:01.309923 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:01.309793    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddcb2e1b-qbrbt" event={"ID":"a90f40d0-8846-469d-b3e9-c9158d667dfa","Type":"ContainerStarted","Data":"c0ca1f3ae26219e847faaaa464ff9c66d26715d67e8229f0bf8be2aa9b5b4358"}
Oct 30 20:48:01.312098 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:01.312077    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5" event={"ID":"55a2c070-6473-481d-a202-f9fcd1231856","Type":"ContainerStarted","Data":"6ce7e12ba11a676a9836dafd1ab2e7c40725aa5f15ca6adf455ddfdb75720e1c"}
Oct 30 20:48:01.312171 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:01.312115    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5" event={"ID":"55a2c070-6473-481d-a202-f9fcd1231856","Type":"ContainerStarted","Data":"0b9ebdc57202f3a59a73a9bc8456cf75b866714f54e9e3420b1f59e637fdb24b"}
Oct 30 20:48:01.314384 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:01.314362    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv" event={"ID":"4349a44e-61fd-4a25-8548-96b11c3cf4cc","Type":"ContainerStarted","Data":"1acec5824a62533f0c9761f48fdd96ce155bf33284a9d7e594da98da2dead2d8"}
Oct 30 20:48:01.314481 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:01.314398    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv" event={"ID":"4349a44e-61fd-4a25-8548-96b11c3cf4cc","Type":"ContainerStarted","Data":"0132177a676b6cf677275ff3c267cc1e5b5d246a832fa0bf641535de02067048"}
Oct 30 20:48:01.428570 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:01.428510    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3114635a-nvgjq" podStartSLOduration=2.428492105 podStartE2EDuration="2.428492105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:01.424171729 +0000 UTC m=+1293.341193266" watchObservedRunningTime="2026-10-30 20:48:01.428492105 +0000 UTC m=+1293.345513642"
Oct 30 20:48:01.553181 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:01.553110    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv" podStartSLOduration=1.553091204 podStartE2EDuration="1.553091204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:01.54570561 +0000 UTC m=+1293.462727147" watchObservedRunningTime="2026-10-30 20:48:01.553091204 +0000 UTC m=+1293.470112741"
Oct 30 20:48:01.623331 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:01.623230    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"
Oct 30 20:48:01.653563 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:01.653527    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"
Oct 30 20:48:01.670373 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:01.670307    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddcb2e1b-qbrbt" podStartSLOduration=2.670289259 podStartE2EDuration="2.670289259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:01.660925709 +0000 UTC m=+1293.577947246" watchObservedRunningTime="2026-10-30 20:48:01.670289259 +0000 UTC m=+1293.587310795"
Oct 30 20:48:01.719850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:01.714496    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51daf491-bb7cc"]
Oct 30 20:48:01.727864 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:01.724992    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51daf491-bb7cc"
Oct 30 20:48:01.795978 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:01.795907    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51daf491-bb7cc"]
Oct 30 20:48:01.799705 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:01.799649    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5" podStartSLOduration=1.79963001 podStartE2EDuration="1.79963001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:01.79084021 +0000 UTC m=+1293.707861747" watchObservedRunningTime="2026-10-30 20:48:01.79963001 +0000 UTC m=+1293.716651547"
Oct 30 20:48:01.812788 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:01.812284    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flsgx\" (UniqueName: \"kubernetes.io/projected/64d24113-5574-4a4a-9bf3-0c2814605103-kube-api-access-flsgx\") pod \"maas-api-revoke-sub-51daf491-bb7cc\" (UID: \"64d24113-5574-4a4a-9bf3-0c2814605103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51daf491-bb7cc"
Oct 30 20:48:01.812788 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:01.812388    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d24113-5574-4a4a-9bf3-0c2814605103-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51daf491-bb7cc\" (UID: \"64d24113-5574-4a4a-9bf3-0c2814605103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51daf491-bb7cc"
Oct 30 20:48:01.812788 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:01.812510    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d24113-5574-4a4a-9bf3-0c2814605103-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51daf491-bb7cc\" (UID: \"64d24113-5574-4a4a-9bf3-0c2814605103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51daf491-bb7cc"
Oct 30 20:48:01.914255 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:01.914119    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flsgx\" (UniqueName: \"kubernetes.io/projected/64d24113-5574-4a4a-9bf3-0c2814605103-kube-api-access-flsgx\") pod \"maas-api-revoke-sub-51daf491-bb7cc\" (UID: \"64d24113-5574-4a4a-9bf3-0c2814605103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51daf491-bb7cc"
Oct 30 20:48:01.915559 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:01.914725    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d24113-5574-4a4a-9bf3-0c2814605103-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51daf491-bb7cc\" (UID: \"64d24113-5574-4a4a-9bf3-0c2814605103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51daf491-bb7cc"
Oct 30 20:48:01.915730 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:01.915564    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d24113-5574-4a4a-9bf3-0c2814605103-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51daf491-bb7cc\" (UID: \"64d24113-5574-4a4a-9bf3-0c2814605103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51daf491-bb7cc"
Oct 30 20:48:01.916620 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:01.916560    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d24113-5574-4a4a-9bf3-0c2814605103-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51daf491-bb7cc\" (UID: \"64d24113-5574-4a4a-9bf3-0c2814605103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51daf491-bb7cc"
Oct 30 20:48:01.922926 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:01.922905    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d24113-5574-4a4a-9bf3-0c2814605103-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51daf491-bb7cc\" (UID: \"64d24113-5574-4a4a-9bf3-0c2814605103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51daf491-bb7cc"
Oct 30 20:48:02.002713 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:02.002673    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flsgx\" (UniqueName: \"kubernetes.io/projected/64d24113-5574-4a4a-9bf3-0c2814605103-kube-api-access-flsgx\") pod \"maas-api-revoke-sub-51daf491-bb7cc\" (UID: \"64d24113-5574-4a4a-9bf3-0c2814605103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51daf491-bb7cc"
Oct 30 20:48:02.097359 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:02.097323    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51daf491-bb7cc"
Oct 30 20:48:02.373861 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:02.371450    5071 generic.go:356] "Generic (PLEG): container finished" podID="cb2c533f-9768-436d-bddc-4de7e46885ec" containerID="b7f3f344358e8524aeeac5a775d512042c711dcd3691fb4fc4a5174ca24cc58f" exitCode=0
Oct 30 20:48:02.373861 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:02.371562    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3114635a-nvgjq" event={"ID":"cb2c533f-9768-436d-bddc-4de7e46885ec","Type":"ContainerDied","Data":"b7f3f344358e8524aeeac5a775d512042c711dcd3691fb4fc4a5174ca24cc58f"}
Oct 30 20:48:02.374130 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:02.373987    5071 generic.go:356] "Generic (PLEG): container finished" podID="a90f40d0-8846-469d-b3e9-c9158d667dfa" containerID="f37ccb8ca15b318706c0c7fd8e5bd6142730a5c4712156d5fa0a73f0ba1ff44f" exitCode=0
Oct 30 20:48:02.387851 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:02.374516    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddcb2e1b-qbrbt" event={"ID":"a90f40d0-8846-469d-b3e9-c9158d667dfa","Type":"ContainerDied","Data":"f37ccb8ca15b318706c0c7fd8e5bd6142730a5c4712156d5fa0a73f0ba1ff44f"}
Oct 30 20:48:02.597671 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:48:02.597618    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64d24113_5574_4a4a_9bf3_0c2814605103.slice/crio-595212b7ad6a6edf644bdc883e705a621faa1b78b7d9944b4d57b49daafe28d8 WatchSource:0}: Error finding container 595212b7ad6a6edf644bdc883e705a621faa1b78b7d9944b4d57b49daafe28d8: Status 404 returned error can't find the container with id 595212b7ad6a6edf644bdc883e705a621faa1b78b7d9944b4d57b49daafe28d8
Oct 30 20:48:02.598973 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:02.598861    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51daf491-bb7cc"]
Oct 30 20:48:02.624483 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:02.624457    5071 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 20:48:02.624579 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:02.624503    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5" podUID="55a2c070-6473-481d-a202-f9fcd1231856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 20:48:02.697992 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:02.697943    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:02.701079 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:02.701050    5071 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 20:48:02.701295 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:02.701261    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv" podUID="4349a44e-61fd-4a25-8548-96b11c3cf4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 20:48:03.407430 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:03.407375    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51daf491-bb7cc" event={"ID":"64d24113-5574-4a4a-9bf3-0c2814605103","Type":"ContainerStarted","Data":"1ffb0dee5cf453e0d5e975979e60c7fffe1d165a05f0126f22a9b3c8030548ec"}
Oct 30 20:48:03.407731 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:03.407695    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51daf491-bb7cc" event={"ID":"64d24113-5574-4a4a-9bf3-0c2814605103","Type":"ContainerStarted","Data":"595212b7ad6a6edf644bdc883e705a621faa1b78b7d9944b4d57b49daafe28d8"}
Oct 30 20:48:03.551961 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:03.527898    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51daf491-bb7cc" podStartSLOduration=2.527875548 podStartE2EDuration="2.527875548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:03.523226046 +0000 UTC m=+1295.440247581" watchObservedRunningTime="2026-10-30 20:48:03.527875548 +0000 UTC m=+1295.444897088"
Oct 30 20:48:03.623805 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:03.623219    5071 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 20:48:03.623805 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:03.623299    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5" podUID="55a2c070-6473-481d-a202-f9fcd1231856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 20:48:03.694882 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:03.694188    5071 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 20:48:03.694882 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:03.694257    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv" podUID="4349a44e-61fd-4a25-8548-96b11c3cf4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 20:48:04.099848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:04.089051    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3114635a-nvgjq"
Oct 30 20:48:04.101880 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:04.101041    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddcb2e1b-qbrbt"
Oct 30 20:48:04.226782 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:04.226722    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb2c533f-9768-436d-bddc-4de7e46885ec-kube-api-access-s7cb9\" (UniqueName: \"kubernetes.io/projected/cb2c533f-9768-436d-bddc-4de7e46885ec-kube-api-access-s7cb9\") pod \"cb2c533f-9768-436d-bddc-4de7e46885ec\" (UID: \"cb2c533f-9768-436d-bddc-4de7e46885ec\") "
Oct 30 20:48:04.226938 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:04.226850    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb2c533f-9768-436d-bddc-4de7e46885ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb2c533f-9768-436d-bddc-4de7e46885ec-maas-api-openshift-ca\") pod \"cb2c533f-9768-436d-bddc-4de7e46885ec\" (UID: \"cb2c533f-9768-436d-bddc-4de7e46885ec\") "
Oct 30 20:48:04.226938 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:04.226912    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a90f40d0-8846-469d-b3e9-c9158d667dfa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a90f40d0-8846-469d-b3e9-c9158d667dfa-maas-api-xks-ca\") pod \"a90f40d0-8846-469d-b3e9-c9158d667dfa\" (UID: \"a90f40d0-8846-469d-b3e9-c9158d667dfa\") "
Oct 30 20:48:04.227033 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:04.227018    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a90f40d0-8846-469d-b3e9-c9158d667dfa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a90f40d0-8846-469d-b3e9-c9158d667dfa-maas-api-openshift-ca\") pod \"a90f40d0-8846-469d-b3e9-c9158d667dfa\" (UID: \"a90f40d0-8846-469d-b3e9-c9158d667dfa\") "
Oct 30 20:48:04.227425 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:04.227160    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a90f40d0-8846-469d-b3e9-c9158d667dfa-kube-api-access-c8m4l\" (UniqueName: \"kubernetes.io/projected/a90f40d0-8846-469d-b3e9-c9158d667dfa-kube-api-access-c8m4l\") pod \"a90f40d0-8846-469d-b3e9-c9158d667dfa\" (UID: \"a90f40d0-8846-469d-b3e9-c9158d667dfa\") "
Oct 30 20:48:04.227425 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:04.227297    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb2c533f-9768-436d-bddc-4de7e46885ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb2c533f-9768-436d-bddc-4de7e46885ec-maas-api-xks-ca\") pod \"cb2c533f-9768-436d-bddc-4de7e46885ec\" (UID: \"cb2c533f-9768-436d-bddc-4de7e46885ec\") "
Oct 30 20:48:04.227425 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:04.227415    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb2c533f-9768-436d-bddc-4de7e46885ec-maas-api-openshift-ca" pod "cb2c533f-9768-436d-bddc-4de7e46885ec" (UID: "cb2c533f-9768-436d-bddc-4de7e46885ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:48:04.227758 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:04.227501    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a90f40d0-8846-469d-b3e9-c9158d667dfa-maas-api-openshift-ca" pod "a90f40d0-8846-469d-b3e9-c9158d667dfa" (UID: "a90f40d0-8846-469d-b3e9-c9158d667dfa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:48:04.258846 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:04.235907    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb2c533f-9768-436d-bddc-4de7e46885ec-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:04.258846 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:04.235939    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a90f40d0-8846-469d-b3e9-c9158d667dfa-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:04.290946 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:04.284480    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb2c533f-9768-436d-bddc-4de7e46885ec-maas-api-xks-ca" pod "cb2c533f-9768-436d-bddc-4de7e46885ec" (UID: "cb2c533f-9768-436d-bddc-4de7e46885ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:48:04.291337 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:04.291297    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a90f40d0-8846-469d-b3e9-c9158d667dfa-maas-api-xks-ca" pod "a90f40d0-8846-469d-b3e9-c9158d667dfa" (UID: "a90f40d0-8846-469d-b3e9-c9158d667dfa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:48:04.291527 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:04.291501    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb2c533f-9768-436d-bddc-4de7e46885ec-kube-api-access-s7cb9" pod "cb2c533f-9768-436d-bddc-4de7e46885ec" (UID: "cb2c533f-9768-436d-bddc-4de7e46885ec"). InnerVolumeSpecName "kube-api-access-s7cb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:48:04.293790 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:04.293757    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a90f40d0-8846-469d-b3e9-c9158d667dfa-kube-api-access-c8m4l" pod "a90f40d0-8846-469d-b3e9-c9158d667dfa" (UID: "a90f40d0-8846-469d-b3e9-c9158d667dfa"). InnerVolumeSpecName "kube-api-access-c8m4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:48:04.362897 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:04.362403    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a90f40d0-8846-469d-b3e9-c9158d667dfa-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:04.362897 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:04.362444    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8m4l\" (UniqueName: \"kubernetes.io/projected/a90f40d0-8846-469d-b3e9-c9158d667dfa-kube-api-access-c8m4l\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:04.362897 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:04.362461    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb2c533f-9768-436d-bddc-4de7e46885ec-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:04.362897 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:04.362478    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7cb9\" (UniqueName: \"kubernetes.io/projected/cb2c533f-9768-436d-bddc-4de7e46885ec-kube-api-access-s7cb9\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:04.418542 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:04.418508    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3114635a-nvgjq" event={"ID":"cb2c533f-9768-436d-bddc-4de7e46885ec","Type":"ContainerDied","Data":"3eb200531e64267e658dd8756dcfc7b3e3b1c7c698cb0951f14773b76fdd6272"}
Oct 30 20:48:04.418735 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:04.418577    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3eb200531e64267e658dd8756dcfc7b3e3b1c7c698cb0951f14773b76fdd6272"
Oct 30 20:48:04.418735 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:04.418673    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3114635a-nvgjq"
Oct 30 20:48:04.423168 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:04.423130    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddcb2e1b-qbrbt"
Oct 30 20:48:04.423295 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:04.423217    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddcb2e1b-qbrbt" event={"ID":"a90f40d0-8846-469d-b3e9-c9158d667dfa","Type":"ContainerDied","Data":"c0ca1f3ae26219e847faaaa464ff9c66d26715d67e8229f0bf8be2aa9b5b4358"}
Oct 30 20:48:04.423295 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:04.423254    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0ca1f3ae26219e847faaaa464ff9c66d26715d67e8229f0bf8be2aa9b5b4358"
Oct 30 20:48:04.424847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:04.424809    5071 generic.go:356] "Generic (PLEG): container finished" podID="64d24113-5574-4a4a-9bf3-0c2814605103" containerID="1ffb0dee5cf453e0d5e975979e60c7fffe1d165a05f0126f22a9b3c8030548ec" exitCode=0
Oct 30 20:48:04.424970 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:04.424850    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51daf491-bb7cc" event={"ID":"64d24113-5574-4a4a-9bf3-0c2814605103","Type":"ContainerDied","Data":"1ffb0dee5cf453e0d5e975979e60c7fffe1d165a05f0126f22a9b3c8030548ec"}
Oct 30 20:48:04.623911 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:04.623789    5071 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 20:48:04.624390 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:04.623919    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5" podUID="55a2c070-6473-481d-a202-f9fcd1231856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 20:48:04.658856 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:04.655202    5071 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 20:48:04.658856 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:04.655256    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv" podUID="4349a44e-61fd-4a25-8548-96b11c3cf4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 20:48:05.655848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:05.623510    5071 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 20:48:05.655848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:05.623616    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5" podUID="55a2c070-6473-481d-a202-f9fcd1231856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 20:48:05.656454 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:05.656193    5071 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 20:48:05.656454 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:05.656248    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv" podUID="4349a44e-61fd-4a25-8548-96b11c3cf4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 20:48:05.796903 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:05.796880    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51daf491-bb7cc"
Oct 30 20:48:05.821887 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:05.821857    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64d24113-5574-4a4a-9bf3-0c2814605103-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d24113-5574-4a4a-9bf3-0c2814605103-maas-api-xks-ca\") pod \"64d24113-5574-4a4a-9bf3-0c2814605103\" (UID: \"64d24113-5574-4a4a-9bf3-0c2814605103\") "
Oct 30 20:48:05.822009 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:05.821892    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64d24113-5574-4a4a-9bf3-0c2814605103-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d24113-5574-4a4a-9bf3-0c2814605103-maas-api-openshift-ca\") pod \"64d24113-5574-4a4a-9bf3-0c2814605103\" (UID: \"64d24113-5574-4a4a-9bf3-0c2814605103\") "
Oct 30 20:48:05.822009 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:05.821955    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64d24113-5574-4a4a-9bf3-0c2814605103-kube-api-access-flsgx\" (UniqueName: \"kubernetes.io/projected/64d24113-5574-4a4a-9bf3-0c2814605103-kube-api-access-flsgx\") pod \"64d24113-5574-4a4a-9bf3-0c2814605103\" (UID: \"64d24113-5574-4a4a-9bf3-0c2814605103\") "
Oct 30 20:48:05.822298 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:05.822274    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64d24113-5574-4a4a-9bf3-0c2814605103-maas-api-openshift-ca" pod "64d24113-5574-4a4a-9bf3-0c2814605103" (UID: "64d24113-5574-4a4a-9bf3-0c2814605103"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:48:05.822633 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:05.822610    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d24113-5574-4a4a-9bf3-0c2814605103-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:05.824190 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:05.824161    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64d24113-5574-4a4a-9bf3-0c2814605103-maas-api-xks-ca" pod "64d24113-5574-4a4a-9bf3-0c2814605103" (UID: "64d24113-5574-4a4a-9bf3-0c2814605103"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:48:05.824341 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:05.824322    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64d24113-5574-4a4a-9bf3-0c2814605103-kube-api-access-flsgx" pod "64d24113-5574-4a4a-9bf3-0c2814605103" (UID: "64d24113-5574-4a4a-9bf3-0c2814605103"). InnerVolumeSpecName "kube-api-access-flsgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:48:05.927293 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:05.927218    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flsgx\" (UniqueName: \"kubernetes.io/projected/64d24113-5574-4a4a-9bf3-0c2814605103-kube-api-access-flsgx\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:05.927293 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:05.927259    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d24113-5574-4a4a-9bf3-0c2814605103-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:06.180906 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:06.179011    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"
Oct 30 20:48:06.579593 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:06.578319    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51daf491-bb7cc" event={"ID":"64d24113-5574-4a4a-9bf3-0c2814605103","Type":"ContainerDied","Data":"595212b7ad6a6edf644bdc883e705a621faa1b78b7d9944b4d57b49daafe28d8"}
Oct 30 20:48:06.579593 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:06.578389    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="595212b7ad6a6edf644bdc883e705a621faa1b78b7d9944b4d57b49daafe28d8"
Oct 30 20:48:06.579593 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:06.578472    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51daf491-bb7cc"
Oct 30 20:48:06.623665 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:06.623617    5071 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 20:48:06.623845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:06.623696    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5" podUID="55a2c070-6473-481d-a202-f9fcd1231856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 20:48:06.674849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:06.654357    5071 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 20:48:06.674849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:06.654415    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv" podUID="4349a44e-61fd-4a25-8548-96b11c3cf4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 20:48:07.624087 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:07.623992    5071 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 20:48:07.624087 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:07.624045    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5" podUID="55a2c070-6473-481d-a202-f9fcd1231856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 20:48:07.654000 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:07.653958    5071 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 30 20:48:07.654191 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:07.654020    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv" podUID="4349a44e-61fd-4a25-8548-96b11c3cf4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 30 20:48:07.872976 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:07.872930    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:08.235196 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:08.235157    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:08.794599 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:08.794563    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"
Oct 30 20:48:08.795620 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:08.795597    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"
Oct 30 20:48:08.796788 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:08.796761    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"
Oct 30 20:48:08.828577 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:08.828549    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"
Oct 30 20:48:08.829806 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:08.829784    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"
Oct 30 20:48:08.831726 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:08.831693    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"
Oct 30 20:48:10.224288 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.224259    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f60b94db-5tt7f"]
Oct 30 20:48:10.225224 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.225202    5071 cpu_manager.go:414] "containerMap: removing container" podUID="cb2c533f-9768-436d-bddc-4de7e46885ec" containerName="revoke-keys"
Oct 30 20:48:10.225280 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.225227    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb2c533f-9768-436d-bddc-4de7e46885ec" containerName="revoke-keys"
Oct 30 20:48:10.225280 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.225258    5071 cpu_manager.go:414] "containerMap: removing container" podUID="a90f40d0-8846-469d-b3e9-c9158d667dfa" containerName="revoke-keys"
Oct 30 20:48:10.225280 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.225264    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a90f40d0-8846-469d-b3e9-c9158d667dfa" containerName="revoke-keys"
Oct 30 20:48:10.225377 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.225287    5071 cpu_manager.go:414] "containerMap: removing container" podUID="64d24113-5574-4a4a-9bf3-0c2814605103" containerName="revoke-keys"
Oct 30 20:48:10.225377 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.225291    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64d24113-5574-4a4a-9bf3-0c2814605103" containerName="revoke-keys"
Oct 30 20:48:10.225452 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.225398    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="64d24113-5574-4a4a-9bf3-0c2814605103" containerName="revoke-keys"
Oct 30 20:48:10.225452 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.225409    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb2c533f-9768-436d-bddc-4de7e46885ec" containerName="revoke-keys"
Oct 30 20:48:10.225452 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.225422    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="a90f40d0-8846-469d-b3e9-c9158d667dfa" containerName="revoke-keys"
Oct 30 20:48:10.230404 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.230377    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f60b94db-5tt7f"
Oct 30 20:48:10.287385 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.287351    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8l4g\" (UniqueName: \"kubernetes.io/projected/0ac57fcd-364e-4d13-a6f9-0308d322bc32-kube-api-access-d8l4g\") pod \"maas-api-revoke-sub-f60b94db-5tt7f\" (UID: \"0ac57fcd-364e-4d13-a6f9-0308d322bc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f60b94db-5tt7f"
Oct 30 20:48:10.288151 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.288125    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ac57fcd-364e-4d13-a6f9-0308d322bc32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f60b94db-5tt7f\" (UID: \"0ac57fcd-364e-4d13-a6f9-0308d322bc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f60b94db-5tt7f"
Oct 30 20:48:10.288309 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.288168    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ac57fcd-364e-4d13-a6f9-0308d322bc32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f60b94db-5tt7f\" (UID: \"0ac57fcd-364e-4d13-a6f9-0308d322bc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f60b94db-5tt7f"
Oct 30 20:48:10.292381 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.292354    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f60b94db-5tt7f"]
Oct 30 20:48:10.334685 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.334657    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:48:10.334904 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.334718    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 20:48:10.389931 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.389892    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8l4g\" (UniqueName: \"kubernetes.io/projected/0ac57fcd-364e-4d13-a6f9-0308d322bc32-kube-api-access-d8l4g\") pod \"maas-api-revoke-sub-f60b94db-5tt7f\" (UID: \"0ac57fcd-364e-4d13-a6f9-0308d322bc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f60b94db-5tt7f"
Oct 30 20:48:10.390103 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.390015    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ac57fcd-364e-4d13-a6f9-0308d322bc32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f60b94db-5tt7f\" (UID: \"0ac57fcd-364e-4d13-a6f9-0308d322bc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f60b94db-5tt7f"
Oct 30 20:48:10.390103 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.390041    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ac57fcd-364e-4d13-a6f9-0308d322bc32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f60b94db-5tt7f\" (UID: \"0ac57fcd-364e-4d13-a6f9-0308d322bc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f60b94db-5tt7f"
Oct 30 20:48:10.390641 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.390609    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ac57fcd-364e-4d13-a6f9-0308d322bc32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f60b94db-5tt7f\" (UID: \"0ac57fcd-364e-4d13-a6f9-0308d322bc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f60b94db-5tt7f"
Oct 30 20:48:10.392873 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.392844    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ac57fcd-364e-4d13-a6f9-0308d322bc32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f60b94db-5tt7f\" (UID: \"0ac57fcd-364e-4d13-a6f9-0308d322bc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f60b94db-5tt7f"
Oct 30 20:48:10.448325 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.448299    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8l4g\" (UniqueName: \"kubernetes.io/projected/0ac57fcd-364e-4d13-a6f9-0308d322bc32-kube-api-access-d8l4g\") pod \"maas-api-revoke-sub-f60b94db-5tt7f\" (UID: \"0ac57fcd-364e-4d13-a6f9-0308d322bc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f60b94db-5tt7f"
Oct 30 20:48:10.460288 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.456250    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a4c3808-fzrrm"]
Oct 30 20:48:10.464525 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.464497    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4c3808-fzrrm"
Oct 30 20:48:10.518689 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.518654    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a4c3808-fzrrm"]
Oct 30 20:48:10.544983 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.544954    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f60b94db-5tt7f"
Oct 30 20:48:10.592423 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.592385    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxpth\" (UniqueName: \"kubernetes.io/projected/cf0d87d0-a8e9-41a4-ae83-18479e8aadd8-kube-api-access-qxpth\") pod \"maas-api-revoke-sub-3a4c3808-fzrrm\" (UID: \"cf0d87d0-a8e9-41a4-ae83-18479e8aadd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4c3808-fzrrm"
Oct 30 20:48:10.592591 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.592456    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf0d87d0-a8e9-41a4-ae83-18479e8aadd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a4c3808-fzrrm\" (UID: \"cf0d87d0-a8e9-41a4-ae83-18479e8aadd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4c3808-fzrrm"
Oct 30 20:48:10.592591 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.592561    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf0d87d0-a8e9-41a4-ae83-18479e8aadd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a4c3808-fzrrm\" (UID: \"cf0d87d0-a8e9-41a4-ae83-18479e8aadd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4c3808-fzrrm"
Oct 30 20:48:10.693816 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.693776    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf0d87d0-a8e9-41a4-ae83-18479e8aadd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a4c3808-fzrrm\" (UID: \"cf0d87d0-a8e9-41a4-ae83-18479e8aadd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4c3808-fzrrm"
Oct 30 20:48:10.694178 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.694119    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxpth\" (UniqueName: \"kubernetes.io/projected/cf0d87d0-a8e9-41a4-ae83-18479e8aadd8-kube-api-access-qxpth\") pod \"maas-api-revoke-sub-3a4c3808-fzrrm\" (UID: \"cf0d87d0-a8e9-41a4-ae83-18479e8aadd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4c3808-fzrrm"
Oct 30 20:48:10.694590 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.694496    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf0d87d0-a8e9-41a4-ae83-18479e8aadd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a4c3808-fzrrm\" (UID: \"cf0d87d0-a8e9-41a4-ae83-18479e8aadd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4c3808-fzrrm"
Oct 30 20:48:10.695555 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.695524    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf0d87d0-a8e9-41a4-ae83-18479e8aadd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a4c3808-fzrrm\" (UID: \"cf0d87d0-a8e9-41a4-ae83-18479e8aadd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4c3808-fzrrm"
Oct 30 20:48:10.697923 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.697843    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf0d87d0-a8e9-41a4-ae83-18479e8aadd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a4c3808-fzrrm\" (UID: \"cf0d87d0-a8e9-41a4-ae83-18479e8aadd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4c3808-fzrrm"
Oct 30 20:48:10.752939 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.752868    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxpth\" (UniqueName: \"kubernetes.io/projected/cf0d87d0-a8e9-41a4-ae83-18479e8aadd8-kube-api-access-qxpth\") pod \"maas-api-revoke-sub-3a4c3808-fzrrm\" (UID: \"cf0d87d0-a8e9-41a4-ae83-18479e8aadd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4c3808-fzrrm"
Oct 30 20:48:10.788996 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.788967    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4c3808-fzrrm"
Oct 30 20:48:10.954929 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:10.954897    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f60b94db-5tt7f"]
Oct 30 20:48:10.957786 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:48:10.957752    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ac57fcd_364e_4d13_a6f9_0308d322bc32.slice/crio-0b7784bb0a138bca36de47588f9e1410466688494ad37e739b309e95ee1b341c WatchSource:0}: Error finding container 0b7784bb0a138bca36de47588f9e1410466688494ad37e739b309e95ee1b341c: Status 404 returned error can't find the container with id 0b7784bb0a138bca36de47588f9e1410466688494ad37e739b309e95ee1b341c
Oct 30 20:48:11.175145 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:11.175118    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a4c3808-fzrrm"]
Oct 30 20:48:11.636185 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:11.636147    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4c3808-fzrrm" event={"ID":"cf0d87d0-a8e9-41a4-ae83-18479e8aadd8","Type":"ContainerStarted","Data":"0d6e686b847f788e45f87077c3140994af323c3366af4e60df295b7fcaed3b06"}
Oct 30 20:48:11.636639 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:11.636210    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4c3808-fzrrm" event={"ID":"cf0d87d0-a8e9-41a4-ae83-18479e8aadd8","Type":"ContainerStarted","Data":"1df6507f72e7fc84e6d5c28f7f198740ad71b54c0c6dfe6a1c346c486a54e31b"}
Oct 30 20:48:11.638253 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:11.638226    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f60b94db-5tt7f" event={"ID":"0ac57fcd-364e-4d13-a6f9-0308d322bc32","Type":"ContainerStarted","Data":"4d82c0b625f6b06529653739fbe13d46973f9437e72b272099b1e795d5943eb3"}
Oct 30 20:48:11.638387 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:11.638278    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f60b94db-5tt7f" event={"ID":"0ac57fcd-364e-4d13-a6f9-0308d322bc32","Type":"ContainerStarted","Data":"0b7784bb0a138bca36de47588f9e1410466688494ad37e739b309e95ee1b341c"}
Oct 30 20:48:11.755272 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:11.755220    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4c3808-fzrrm" podStartSLOduration=1.755203586 podStartE2EDuration="1.755203586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:11.75128609 +0000 UTC m=+1303.668307626" watchObservedRunningTime="2026-10-30 20:48:11.755203586 +0000 UTC m=+1303.672225122"
Oct 30 20:48:11.884520 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:11.884352    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f60b94db-5tt7f" podStartSLOduration=1.884337798 podStartE2EDuration="1.884337798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:11.883398241 +0000 UTC m=+1303.800419777" watchObservedRunningTime="2026-10-30 20:48:11.884337798 +0000 UTC m=+1303.801359333"
Oct 30 20:48:12.648443 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:12.648400    5071 generic.go:356] "Generic (PLEG): container finished" podID="cf0d87d0-a8e9-41a4-ae83-18479e8aadd8" containerID="0d6e686b847f788e45f87077c3140994af323c3366af4e60df295b7fcaed3b06" exitCode=0
Oct 30 20:48:12.648909 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:12.648486    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4c3808-fzrrm" event={"ID":"cf0d87d0-a8e9-41a4-ae83-18479e8aadd8","Type":"ContainerDied","Data":"0d6e686b847f788e45f87077c3140994af323c3366af4e60df295b7fcaed3b06"}
Oct 30 20:48:12.650834 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:12.650787    5071 generic.go:356] "Generic (PLEG): container finished" podID="0ac57fcd-364e-4d13-a6f9-0308d322bc32" containerID="4d82c0b625f6b06529653739fbe13d46973f9437e72b272099b1e795d5943eb3" exitCode=0
Oct 30 20:48:12.650963 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:12.650853    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f60b94db-5tt7f" event={"ID":"0ac57fcd-364e-4d13-a6f9-0308d322bc32","Type":"ContainerDied","Data":"4d82c0b625f6b06529653739fbe13d46973f9437e72b272099b1e795d5943eb3"}
Oct 30 20:48:12.926931 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:12.924938    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:13.850075 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:13.850043    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f60b94db-5tt7f"
Oct 30 20:48:13.892473 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:13.892445    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4c3808-fzrrm"
Oct 30 20:48:13.961365 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:13.961281    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ac57fcd-364e-4d13-a6f9-0308d322bc32-kube-api-access-d8l4g\" (UniqueName: \"kubernetes.io/projected/0ac57fcd-364e-4d13-a6f9-0308d322bc32-kube-api-access-d8l4g\") pod \"0ac57fcd-364e-4d13-a6f9-0308d322bc32\" (UID: \"0ac57fcd-364e-4d13-a6f9-0308d322bc32\") "
Oct 30 20:48:13.961525 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:13.961398    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ac57fcd-364e-4d13-a6f9-0308d322bc32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ac57fcd-364e-4d13-a6f9-0308d322bc32-maas-api-openshift-ca\") pod \"0ac57fcd-364e-4d13-a6f9-0308d322bc32\" (UID: \"0ac57fcd-364e-4d13-a6f9-0308d322bc32\") "
Oct 30 20:48:13.961585 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:13.961551    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ac57fcd-364e-4d13-a6f9-0308d322bc32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ac57fcd-364e-4d13-a6f9-0308d322bc32-maas-api-xks-ca\") pod \"0ac57fcd-364e-4d13-a6f9-0308d322bc32\" (UID: \"0ac57fcd-364e-4d13-a6f9-0308d322bc32\") "
Oct 30 20:48:13.961779 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:13.961747    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ac57fcd-364e-4d13-a6f9-0308d322bc32-maas-api-openshift-ca" pod "0ac57fcd-364e-4d13-a6f9-0308d322bc32" (UID: "0ac57fcd-364e-4d13-a6f9-0308d322bc32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:48:13.962399 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:13.962376    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ac57fcd-364e-4d13-a6f9-0308d322bc32-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:13.963691 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:13.963669    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ac57fcd-364e-4d13-a6f9-0308d322bc32-maas-api-xks-ca" pod "0ac57fcd-364e-4d13-a6f9-0308d322bc32" (UID: "0ac57fcd-364e-4d13-a6f9-0308d322bc32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:48:13.963913 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:13.963889    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ac57fcd-364e-4d13-a6f9-0308d322bc32-kube-api-access-d8l4g" pod "0ac57fcd-364e-4d13-a6f9-0308d322bc32" (UID: "0ac57fcd-364e-4d13-a6f9-0308d322bc32"). InnerVolumeSpecName "kube-api-access-d8l4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:48:14.063160 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:14.063121    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf0d87d0-a8e9-41a4-ae83-18479e8aadd8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf0d87d0-a8e9-41a4-ae83-18479e8aadd8-maas-api-xks-ca\") pod \"cf0d87d0-a8e9-41a4-ae83-18479e8aadd8\" (UID: \"cf0d87d0-a8e9-41a4-ae83-18479e8aadd8\") "
Oct 30 20:48:14.063372 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:14.063236    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf0d87d0-a8e9-41a4-ae83-18479e8aadd8-kube-api-access-qxpth\" (UniqueName: \"kubernetes.io/projected/cf0d87d0-a8e9-41a4-ae83-18479e8aadd8-kube-api-access-qxpth\") pod \"cf0d87d0-a8e9-41a4-ae83-18479e8aadd8\" (UID: \"cf0d87d0-a8e9-41a4-ae83-18479e8aadd8\") "
Oct 30 20:48:14.063372 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:14.063273    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf0d87d0-a8e9-41a4-ae83-18479e8aadd8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf0d87d0-a8e9-41a4-ae83-18479e8aadd8-maas-api-openshift-ca\") pod \"cf0d87d0-a8e9-41a4-ae83-18479e8aadd8\" (UID: \"cf0d87d0-a8e9-41a4-ae83-18479e8aadd8\") "
Oct 30 20:48:14.063799 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:14.063768    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf0d87d0-a8e9-41a4-ae83-18479e8aadd8-maas-api-openshift-ca" pod "cf0d87d0-a8e9-41a4-ae83-18479e8aadd8" (UID: "cf0d87d0-a8e9-41a4-ae83-18479e8aadd8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:48:14.064155 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:14.064091    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ac57fcd-364e-4d13-a6f9-0308d322bc32-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:14.064155 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:14.064108    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8l4g\" (UniqueName: \"kubernetes.io/projected/0ac57fcd-364e-4d13-a6f9-0308d322bc32-kube-api-access-d8l4g\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:14.064155 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:14.064118    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf0d87d0-a8e9-41a4-ae83-18479e8aadd8-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:14.065561 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:14.065537    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf0d87d0-a8e9-41a4-ae83-18479e8aadd8-maas-api-xks-ca" pod "cf0d87d0-a8e9-41a4-ae83-18479e8aadd8" (UID: "cf0d87d0-a8e9-41a4-ae83-18479e8aadd8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:48:14.065787 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:14.065767    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf0d87d0-a8e9-41a4-ae83-18479e8aadd8-kube-api-access-qxpth" pod "cf0d87d0-a8e9-41a4-ae83-18479e8aadd8" (UID: "cf0d87d0-a8e9-41a4-ae83-18479e8aadd8"). InnerVolumeSpecName "kube-api-access-qxpth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:48:14.165073 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:14.165039    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf0d87d0-a8e9-41a4-ae83-18479e8aadd8-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:14.165073 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:14.165070    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxpth\" (UniqueName: \"kubernetes.io/projected/cf0d87d0-a8e9-41a4-ae83-18479e8aadd8-kube-api-access-qxpth\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:14.669217 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:14.669184    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4c3808-fzrrm"
Oct 30 20:48:14.669395 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:14.669188    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4c3808-fzrrm" event={"ID":"cf0d87d0-a8e9-41a4-ae83-18479e8aadd8","Type":"ContainerDied","Data":"1df6507f72e7fc84e6d5c28f7f198740ad71b54c0c6dfe6a1c346c486a54e31b"}
Oct 30 20:48:14.669395 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:14.669311    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1df6507f72e7fc84e6d5c28f7f198740ad71b54c0c6dfe6a1c346c486a54e31b"
Oct 30 20:48:14.670875 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:14.670855    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f60b94db-5tt7f"
Oct 30 20:48:14.671114 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:14.670871    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f60b94db-5tt7f" event={"ID":"0ac57fcd-364e-4d13-a6f9-0308d322bc32","Type":"ContainerDied","Data":"0b7784bb0a138bca36de47588f9e1410466688494ad37e739b309e95ee1b341c"}
Oct 30 20:48:14.671236 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:14.671139    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b7784bb0a138bca36de47588f9e1410466688494ad37e739b309e95ee1b341c"
Oct 30 20:48:15.697945 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:15.697913    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"]
Oct 30 20:48:15.698813 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:15.698789    5071 cpu_manager.go:414] "containerMap: removing container" podUID="0ac57fcd-364e-4d13-a6f9-0308d322bc32" containerName="revoke-keys"
Oct 30 20:48:15.698813 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:15.698811    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ac57fcd-364e-4d13-a6f9-0308d322bc32" containerName="revoke-keys"
Oct 30 20:48:15.699008 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:15.698843    5071 cpu_manager.go:414] "containerMap: removing container" podUID="cf0d87d0-a8e9-41a4-ae83-18479e8aadd8" containerName="revoke-keys"
Oct 30 20:48:15.699008 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:15.698852    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf0d87d0-a8e9-41a4-ae83-18479e8aadd8" containerName="revoke-keys"
Oct 30 20:48:15.699008 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:15.698982    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ac57fcd-364e-4d13-a6f9-0308d322bc32" containerName="revoke-keys"
Oct 30 20:48:15.699008 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:15.698992    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf0d87d0-a8e9-41a4-ae83-18479e8aadd8" containerName="revoke-keys"
Oct 30 20:48:15.702979 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:15.702956    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"
Oct 30 20:48:15.789222 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:15.789191    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/736322c2-4cdb-4fd1-bbc1-15c089d827d9-istiod-ca-cert\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47\" (UID: \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"
Oct 30 20:48:15.789395 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:15.789257    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/736322c2-4cdb-4fd1-bbc1-15c089d827d9-istio-data\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47\" (UID: \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"
Oct 30 20:48:15.789395 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:15.789321    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/736322c2-4cdb-4fd1-bbc1-15c089d827d9-workload-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47\" (UID: \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"
Oct 30 20:48:15.789485 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:15.789403    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/736322c2-4cdb-4fd1-bbc1-15c089d827d9-istio-token\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47\" (UID: \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"
Oct 30 20:48:15.789485 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:15.789435    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/736322c2-4cdb-4fd1-bbc1-15c089d827d9-workload-certs\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47\" (UID: \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"
Oct 30 20:48:15.789485 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:15.789473    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/736322c2-4cdb-4fd1-bbc1-15c089d827d9-istio-envoy\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47\" (UID: \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"
Oct 30 20:48:15.789581 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:15.789562    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4hh9\" (UniqueName: \"kubernetes.io/projected/736322c2-4cdb-4fd1-bbc1-15c089d827d9-kube-api-access-k4hh9\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47\" (UID: \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"
Oct 30 20:48:15.789626 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:15.789596    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/736322c2-4cdb-4fd1-bbc1-15c089d827d9-istio-podinfo\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47\" (UID: \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"
Oct 30 20:48:15.789792 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:15.789767    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/736322c2-4cdb-4fd1-bbc1-15c089d827d9-credential-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47\" (UID: \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"
Oct 30 20:48:15.811325 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:15.811291    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"]
Oct 30 20:48:15.891864 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:15.891804    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/736322c2-4cdb-4fd1-bbc1-15c089d827d9-istio-data\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47\" (UID: \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"
Oct 30 20:48:15.892109 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:15.891899    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/736322c2-4cdb-4fd1-bbc1-15c089d827d9-workload-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47\" (UID: \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"
Oct 30 20:48:15.892109 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:15.891934    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/736322c2-4cdb-4fd1-bbc1-15c089d827d9-istio-token\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47\" (UID: \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"
Oct 30 20:48:15.892109 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:15.891957    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/736322c2-4cdb-4fd1-bbc1-15c089d827d9-workload-certs\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47\" (UID: \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"
Oct 30 20:48:15.892109 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:15.892002    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/736322c2-4cdb-4fd1-bbc1-15c089d827d9-istio-envoy\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47\" (UID: \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"
Oct 30 20:48:15.892109 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:15.892070    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4hh9\" (UniqueName: \"kubernetes.io/projected/736322c2-4cdb-4fd1-bbc1-15c089d827d9-kube-api-access-k4hh9\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47\" (UID: \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"
Oct 30 20:48:15.892390 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:15.892115    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/736322c2-4cdb-4fd1-bbc1-15c089d827d9-istio-podinfo\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47\" (UID: \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"
Oct 30 20:48:15.892390 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:15.892223    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/736322c2-4cdb-4fd1-bbc1-15c089d827d9-credential-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47\" (UID: \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"
Oct 30 20:48:15.892390 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:15.892253    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/736322c2-4cdb-4fd1-bbc1-15c089d827d9-istio-data\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47\" (UID: \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"
Oct 30 20:48:15.892515 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:15.892400    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/736322c2-4cdb-4fd1-bbc1-15c089d827d9-workload-certs\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47\" (UID: \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"
Oct 30 20:48:15.892652 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:15.892624    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/736322c2-4cdb-4fd1-bbc1-15c089d827d9-istiod-ca-cert\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47\" (UID: \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"
Oct 30 20:48:15.892652 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:15.892647    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/736322c2-4cdb-4fd1-bbc1-15c089d827d9-credential-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47\" (UID: \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"
Oct 30 20:48:15.893094 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:15.892793    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/736322c2-4cdb-4fd1-bbc1-15c089d827d9-workload-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47\" (UID: \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"
Oct 30 20:48:15.893218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:15.893194    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/736322c2-4cdb-4fd1-bbc1-15c089d827d9-istiod-ca-cert\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47\" (UID: \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"
Oct 30 20:48:15.894626 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:15.894597    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/736322c2-4cdb-4fd1-bbc1-15c089d827d9-istio-envoy\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47\" (UID: \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"
Oct 30 20:48:15.894976 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:15.894955    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/736322c2-4cdb-4fd1-bbc1-15c089d827d9-istio-podinfo\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47\" (UID: \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"
Oct 30 20:48:15.951113 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:15.951048    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/736322c2-4cdb-4fd1-bbc1-15c089d827d9-istio-token\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47\" (UID: \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"
Oct 30 20:48:15.951322 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:15.951298    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4hh9\" (UniqueName: \"kubernetes.io/projected/736322c2-4cdb-4fd1-bbc1-15c089d827d9-kube-api-access-k4hh9\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47\" (UID: \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"
Oct 30 20:48:16.018422 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:16.018397    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"
Oct 30 20:48:16.471008 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:16.470966    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"]
Oct 30 20:48:16.476274 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:48:16.476231    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod736322c2_4cdb_4fd1_bbc1_15c089d827d9.slice/crio-e49e39bcb7b2d095695e309bf78b3b4b7e4f353718671f50c9b9aad219b84955 WatchSource:0}: Error finding container e49e39bcb7b2d095695e309bf78b3b4b7e4f353718671f50c9b9aad219b84955: Status 404 returned error can't find the container with id e49e39bcb7b2d095695e309bf78b3b4b7e4f353718671f50c9b9aad219b84955
Oct 30 20:48:16.479003 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:16.478965    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:48:16.479131 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:16.479075    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:48:16.479131 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:16.479119    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:48:16.691430 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:16.691398    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47" event={"ID":"736322c2-4cdb-4fd1-bbc1-15c089d827d9","Type":"ContainerStarted","Data":"2d13fba3913f9425b772b991272a49bbf4d1d0054558e2b5b5de3a4159d5fe4c"}
Oct 30 20:48:16.691601 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:16.691449    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47" event={"ID":"736322c2-4cdb-4fd1-bbc1-15c089d827d9","Type":"ContainerStarted","Data":"e49e39bcb7b2d095695e309bf78b3b4b7e4f353718671f50c9b9aad219b84955"}
Oct 30 20:48:16.731035 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:48:16.730941    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:48:16.814965 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:16.814908    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47" podStartSLOduration=1.814890734 podStartE2EDuration="1.814890734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:16.811240898 +0000 UTC m=+1308.728262434" watchObservedRunningTime="2026-10-30 20:48:16.814890734 +0000 UTC m=+1308.731912282"
Oct 30 20:48:17.019924 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:17.019774    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"
Oct 30 20:48:17.021936 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:17.021895    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 20:48:17.022196 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:17.022167    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47" podUID="736322c2-4cdb-4fd1-bbc1-15c089d827d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 20:48:18.014776 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:18.014736    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:18.020961 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:18.020924    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 20:48:18.021205 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:18.021182    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47" podUID="736322c2-4cdb-4fd1-bbc1-15c089d827d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 20:48:19.020384 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:19.020345    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 20:48:19.021020 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:19.020976    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47" podUID="736322c2-4cdb-4fd1-bbc1-15c089d827d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 20:48:19.695731 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:19.695684    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 20:48:19.912023 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:19.911979    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:20.026900 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:20.025322    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 20:48:20.026900 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:20.025435    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47" podUID="736322c2-4cdb-4fd1-bbc1-15c089d827d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 20:48:20.128495 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:20.128458    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:20.141303 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:20.141261    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:20.323538 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:20.323502    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:20.502494 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:20.446100    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:20.502494 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:20.446522    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 20:48:20.714850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:20.714150    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:20.715071 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:20.714996    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:48:21.033097 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:21.033002    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 20:48:21.121293 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:21.073205    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47" podUID="736322c2-4cdb-4fd1-bbc1-15c089d827d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 20:48:21.121293 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:21.039398    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:21.121293 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:21.073911    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:48:21.179277 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:21.179231    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 20:48:21.211686 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:21.179881    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:48:21.645158 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:21.633036    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:21.646856 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:21.646790    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 20:48:21.658615 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:21.658583    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:48:21.658916 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:21.658898    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:48:21.690255 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:21.690214    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:21.778850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:21.714810    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:21.891961 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:21.891918    5071 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:48:21.920911 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:21.895894    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t" podUID="00b1175c-ea5d-49a8-b721-93fd709970c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:22.090849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:22.080290    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:22.144849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:22.119384    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 20:48:22.144849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:22.119446    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47" podUID="736322c2-4cdb-4fd1-bbc1-15c089d827d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 20:48:22.284911 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:22.251906    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:22.284911 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:22.254893    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:48:22.284911 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:22.254954    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk" podUID="6295120b-8bbc-4502-8429-4f04d7d5c5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:22.745848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:22.745041    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 20:48:22.800855 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:22.798402    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:48:22.800855 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:22.798493    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g" podUID="05a72fc0-2554-4f73-935e-ade8fb962a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:22.861849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:22.829223    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 20:48:23.042703 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:23.042667    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 20:48:23.043221 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:23.043187    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47" podUID="736322c2-4cdb-4fd1-bbc1-15c089d827d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 20:48:23.146211 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:23.137729    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 20:48:23.278840 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:23.278768    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:23.505808 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:48:23.504763    5071 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.3s"
Oct 30 20:48:23.525529 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:23.387653    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 30 20:48:23.547170 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:23.547133    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:48:23.547482 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:23.547449    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87" podUID="723f1905-6d5a-45a7-a2bb-d5753a0fda9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:23.598306 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:23.598268    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:48:23.598634 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:23.598577    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42" podUID="c4d634d9-39de-4614-9599-38adf86829e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:23.758016 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:23.757907    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:23.858088 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:23.858047    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 20:48:23.976697 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:23.976657    5071 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:24.043943 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:24.019433    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 20:48:24.043943 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:24.019487    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47" podUID="736322c2-4cdb-4fd1-bbc1-15c089d827d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 20:48:24.061054 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:24.060999    5071 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-v9744 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:48:24.061357 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:24.061305    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744" podUID="84cb8a9e-b4d1-4c1b-9338-2ea77abec886" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:24.258848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:24.256338    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:48:24.284183 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:24.278701    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:24.482299 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:24.482199    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:24.561033 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:24.560991    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:48:24.599952 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:24.599908    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8" podUID="191943aa-3324-4132-9edf-d2fbfc748b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:24.654178 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:24.654135    5071 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:48:24.654501 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:24.654445    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts" podUID="7299979e-94b4-4d26-a860-c88b6136faf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:24.659940 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:24.659909    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:48:24.665899 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:24.660115    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj" podUID="09665d83-c120-4c47-bc5f-45fa16149bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:24.788049 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:24.787992    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:24.856035 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:24.855983    5071 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:48:24.856322 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:24.856278    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5" podUID="55a2c070-6473-481d-a202-f9fcd1231856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:24.859065 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:24.859029    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:24.914969 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:24.914913    5071 patch_prober.go:29] interesting pod/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:48:24.915258 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:24.915221    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv" podUID="4349a44e-61fd-4a25-8548-96b11c3cf4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:25.028339 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:25.028301    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 20:48:25.028612 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:25.028582    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47" podUID="736322c2-4cdb-4fd1-bbc1-15c089d827d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 20:48:25.305880 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:25.301499    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:25.385874 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:25.360446    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:25.385874 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:25.369970    5071 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:48:25.385874 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:25.370024    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz" podUID="203f757a-7115-466c-8a13-14fb326599ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:25.493589 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:25.475014    5071 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:48:25.493589 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:25.475079    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq" podUID="3dbc434f-60ea-4a24-9115-b5db756cbec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:25.493589 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:25.485597    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 20:48:25.813894 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:25.807932    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:25.859901 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:25.859857    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:26.091277 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:26.084730    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 20:48:26.091277 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:26.084794    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47" podUID="736322c2-4cdb-4fd1-bbc1-15c089d827d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 20:48:26.132700 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:26.092363    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:26.305849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:26.305491    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 20:48:26.369421 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:26.369316    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:26.372189 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:26.372164    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:48:26.498180 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:26.498136    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:26.501412 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:26.501369    5071 patch_prober.go:29] interesting pod/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:48:26.501648 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:26.501625    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp" podUID="cefe43d8-48c1-4ba9-923c-3c2da5db8f01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:26.565843 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:26.565785    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"]
Oct 30 20:48:26.826850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:26.820677    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:26.826850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:26.822016    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:26.888069 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:26.888027    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:27.017849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.017791    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:27.049968 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.049920    5071 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": context deadline exceeded"
Oct 30 20:48:27.050544 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.050512    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 20:48:27.050777 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.050753    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47" podUID="736322c2-4cdb-4fd1-bbc1-15c089d827d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 20:48:27.051230 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.051207    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:48:27.110965 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.085786    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-workload-socket\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c\" (UID: \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:48:27.128857 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.119923    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-workload-certs\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c\" (UID: \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:48:27.146806 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.146764    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-credential-socket\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c\" (UID: \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:48:27.147202 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.147177    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-istio-envoy\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c\" (UID: \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:48:27.165020 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.164982    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-istiod-ca-cert\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c\" (UID: \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:48:27.165304 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.165279    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-istio-podinfo\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c\" (UID: \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:48:27.165538 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.165518    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-istio-token\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c\" (UID: \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:48:27.165751 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.165733    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r6sx\" (UniqueName: \"kubernetes.io/projected/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-kube-api-access-7r6sx\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c\" (UID: \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:48:27.168987 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.168960    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-istio-data\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c\" (UID: \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:48:27.231932 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.231898    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-78ca07a3-openshift-default-dockercfg-2f4mk\""
Oct 30 20:48:27.277773 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.265365    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"]
Oct 30 20:48:27.313228 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.313190    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:27.332488 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.332445    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-istio-envoy\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c\" (UID: \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:48:27.363365 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.363331    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-istiod-ca-cert\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c\" (UID: \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:48:27.373918 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.373877    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-istio-podinfo\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c\" (UID: \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:48:27.441747 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.399182    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-istio-token\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c\" (UID: \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:48:27.441747 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.399323    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r6sx\" (UniqueName: \"kubernetes.io/projected/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-kube-api-access-7r6sx\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c\" (UID: \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:48:27.441747 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.399376    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-istio-data\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c\" (UID: \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:48:27.441747 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.399442    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-workload-socket\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c\" (UID: \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:48:27.441747 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.399472    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-workload-certs\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c\" (UID: \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:48:27.441747 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.399556    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-credential-socket\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c\" (UID: \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:48:27.441747 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.397186    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 30 20:48:27.441747 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.399023    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-istiod-ca-cert\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c\" (UID: \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:48:27.441747 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.419488    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-istio-data\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c\" (UID: \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:48:27.441747 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.440091    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-workload-socket\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c\" (UID: \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:48:27.441747 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.440405    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-workload-certs\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c\" (UID: \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:48:27.441747 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.441682    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-credential-socket\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c\" (UID: \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:48:27.552920 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.531134    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:27.563032 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.559614    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-istio-podinfo\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c\" (UID: \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:48:27.641278 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.641196    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-istio-envoy\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c\" (UID: \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:48:27.803844 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.795158    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r6sx\" (UniqueName: \"kubernetes.io/projected/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-kube-api-access-7r6sx\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c\" (UID: \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:48:27.888852 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.870429    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 20:48:27.890495 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.890147    5071 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded"
Oct 30 20:48:27.944521 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:27.944412    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 20:48:28.003989 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:28.003399    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-istio-token\") pod \"e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c\" (UID: \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\") " pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:48:28.095749 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:28.095622    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 20:48:28.095749 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:28.095684    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47" podUID="736322c2-4cdb-4fd1-bbc1-15c089d827d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 20:48:28.608845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:28.599101    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:28.881901 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:28.797062    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 30 20:48:28.881901 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:28.863398    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 20:48:29.015078 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:28.957619    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 20:48:29.047814 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:29.034299    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-78ca07a3-openshift-default-dockercfg-2f4mk\""
Oct 30 20:48:29.048089 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:29.043794    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 20:48:29.069913 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:29.048232    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47" podUID="736322c2-4cdb-4fd1-bbc1-15c089d827d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 20:48:29.073067 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:29.071643    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 20:48:29.113294 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:29.071918    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:48:29.525845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:29.503933    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:29.663571 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:29.613765    5071 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded"
Oct 30 20:48:29.822301 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:29.822243    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:29.874663 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:29.874597    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 20:48:30.044915 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:30.027392    5071 scope.go:122] "RemoveContainer" containerID="d808682681f740430b61931999e763e8e3cd17f32a353aab2563b06172457771"
Oct 30 20:48:30.156896 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:48:30.142116    5071 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.572s"
Oct 30 20:48:30.176689 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:30.176652    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 20:48:30.177013 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:30.176967    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47" podUID="736322c2-4cdb-4fd1-bbc1-15c089d827d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 20:48:30.486895 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:30.486714    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:30.739258 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:30.715418    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:48:30.739258 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:30.738111    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:48:30.739258 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:30.738152    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"]
Oct 30 20:48:30.767214 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:30.767060    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:30.810187 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:30.810155    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"]
Oct 30 20:48:30.810419 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:30.810407    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa138efb-67b2j"]
Oct 30 20:48:30.828771 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:30.828746    5071 scope.go:122] "RemoveContainer" containerID="f111f62c56b575b4e35eb61fa0903f36968c214fb7b213a528e692ecc0c5e648"
Oct 30 20:48:30.846397 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:30.846370    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:48:30.912244 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:30.912218    5071 scope.go:122] "RemoveContainer" containerID="e25e5bafb126a74b31fa521055153a0aa6e480158a57ef92d2a55f0907b8ec28"
Oct 30 20:48:30.914206 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:30.914181    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5944af0-5719-461a-b5bc-0c05023e086d-istio-token\") pod \"e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7\" (UID: \"e5944af0-5719-461a-b5bc-0c05023e086d\") " pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:48:30.914526 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:30.914503    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5944af0-5719-461a-b5bc-0c05023e086d-workload-certs\") pod \"e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7\" (UID: \"e5944af0-5719-461a-b5bc-0c05023e086d\") " pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:48:30.914768 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:30.914747    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5944af0-5719-461a-b5bc-0c05023e086d-istio-data\") pod \"e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7\" (UID: \"e5944af0-5719-461a-b5bc-0c05023e086d\") " pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:48:30.914987 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:30.914966    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tspx\" (UniqueName: \"kubernetes.io/projected/e5944af0-5719-461a-b5bc-0c05023e086d-kube-api-access-5tspx\") pod \"e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7\" (UID: \"e5944af0-5719-461a-b5bc-0c05023e086d\") " pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:48:30.915418 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:30.915144    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5944af0-5719-461a-b5bc-0c05023e086d-istio-envoy\") pod \"e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7\" (UID: \"e5944af0-5719-461a-b5bc-0c05023e086d\") " pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:48:30.915418 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:30.915204    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5944af0-5719-461a-b5bc-0c05023e086d-workload-socket\") pod \"e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7\" (UID: \"e5944af0-5719-461a-b5bc-0c05023e086d\") " pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:48:30.915418 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:30.915253    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5944af0-5719-461a-b5bc-0c05023e086d-istiod-ca-cert\") pod \"e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7\" (UID: \"e5944af0-5719-461a-b5bc-0c05023e086d\") " pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:48:30.915418 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:30.915348    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5944af0-5719-461a-b5bc-0c05023e086d-credential-socket\") pod \"e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7\" (UID: \"e5944af0-5719-461a-b5bc-0c05023e086d\") " pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:48:30.915418 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:30.915388    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5944af0-5719-461a-b5bc-0c05023e086d-istio-podinfo\") pod \"e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7\" (UID: \"e5944af0-5719-461a-b5bc-0c05023e086d\") " pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:48:30.927656 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:30.922376    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-bd92310f-openshift-default-dockercfg-2ljz2\""
Oct 30 20:48:30.972130 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:30.972083    5071 scope.go:122] "RemoveContainer" containerID="97b6f26117d793db59de994ec374fa9dd199723e92fc824cde25c66b304a32f3"
Oct 30 20:48:30.990532 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:30.981098    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:31.019098 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.019055    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa138efb-67b2j"]
Oct 30 20:48:31.019271 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.019161    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:48:31.019271 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.019201    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:48:31.019271 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.019233    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:48:31.019432 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.019280    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"
Oct 30 20:48:31.019970 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.019950    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa138efb-67b2j"
Oct 30 20:48:31.045993 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.020352    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 20:48:31.116346 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.055757    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5944af0-5719-461a-b5bc-0c05023e086d-istio-podinfo\") pod \"e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7\" (UID: \"e5944af0-5719-461a-b5bc-0c05023e086d\") " pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:48:31.116346 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.069433    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5944af0-5719-461a-b5bc-0c05023e086d-istio-token\") pod \"e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7\" (UID: \"e5944af0-5719-461a-b5bc-0c05023e086d\") " pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:48:31.116346 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.069630    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5944af0-5719-461a-b5bc-0c05023e086d-workload-certs\") pod \"e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7\" (UID: \"e5944af0-5719-461a-b5bc-0c05023e086d\") " pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:48:31.116346 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.069677    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5944af0-5719-461a-b5bc-0c05023e086d-istio-data\") pod \"e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7\" (UID: \"e5944af0-5719-461a-b5bc-0c05023e086d\") " pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:48:31.116346 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.069760    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tspx\" (UniqueName: \"kubernetes.io/projected/e5944af0-5719-461a-b5bc-0c05023e086d-kube-api-access-5tspx\") pod \"e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7\" (UID: \"e5944af0-5719-461a-b5bc-0c05023e086d\") " pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:48:31.116346 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.112433    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5944af0-5719-461a-b5bc-0c05023e086d-workload-certs\") pod \"e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7\" (UID: \"e5944af0-5719-461a-b5bc-0c05023e086d\") " pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:48:31.116346 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.069163    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5944af0-5719-461a-b5bc-0c05023e086d-istio-podinfo\") pod \"e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7\" (UID: \"e5944af0-5719-461a-b5bc-0c05023e086d\") " pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:48:31.116346 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.113104    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5944af0-5719-461a-b5bc-0c05023e086d-istio-data\") pod \"e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7\" (UID: \"e5944af0-5719-461a-b5bc-0c05023e086d\") " pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:48:31.196357 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.138886    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5944af0-5719-461a-b5bc-0c05023e086d-istio-envoy\") pod \"e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7\" (UID: \"e5944af0-5719-461a-b5bc-0c05023e086d\") " pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:48:31.196357 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.138979    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5944af0-5719-461a-b5bc-0c05023e086d-workload-socket\") pod \"e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7\" (UID: \"e5944af0-5719-461a-b5bc-0c05023e086d\") " pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:48:31.196357 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.139032    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5944af0-5719-461a-b5bc-0c05023e086d-istiod-ca-cert\") pod \"e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7\" (UID: \"e5944af0-5719-461a-b5bc-0c05023e086d\") " pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:48:31.196357 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.139118    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5944af0-5719-461a-b5bc-0c05023e086d-credential-socket\") pod \"e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7\" (UID: \"e5944af0-5719-461a-b5bc-0c05023e086d\") " pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:48:31.196357 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.139610    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5944af0-5719-461a-b5bc-0c05023e086d-credential-socket\") pod \"e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7\" (UID: \"e5944af0-5719-461a-b5bc-0c05023e086d\") " pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:48:31.196357 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.139902    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5944af0-5719-461a-b5bc-0c05023e086d-workload-socket\") pod \"e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7\" (UID: \"e5944af0-5719-461a-b5bc-0c05023e086d\") " pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:48:31.196357 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.140680    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5944af0-5719-461a-b5bc-0c05023e086d-istiod-ca-cert\") pod \"e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7\" (UID: \"e5944af0-5719-461a-b5bc-0c05023e086d\") " pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:48:31.196357 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.179317    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:48:31.196357 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.179459    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 20:48:31.196357 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.185368    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5944af0-5719-461a-b5bc-0c05023e086d-istio-envoy\") pod \"e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7\" (UID: \"e5944af0-5719-461a-b5bc-0c05023e086d\") " pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:48:31.196357 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.069274    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47" podUID="736322c2-4cdb-4fd1-bbc1-15c089d827d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 20:48:31.261155 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.249932    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkqqh\" (UniqueName: \"kubernetes.io/projected/b913b363-6fbf-4031-a833-2e5e71ef1f19-kube-api-access-bkqqh\") pod \"maas-api-revoke-sub-aa138efb-67b2j\" (UID: \"b913b363-6fbf-4031-a833-2e5e71ef1f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa138efb-67b2j"
Oct 30 20:48:31.261155 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.250018    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b913b363-6fbf-4031-a833-2e5e71ef1f19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa138efb-67b2j\" (UID: \"b913b363-6fbf-4031-a833-2e5e71ef1f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa138efb-67b2j"
Oct 30 20:48:31.296728 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.296644    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5944af0-5719-461a-b5bc-0c05023e086d-istio-token\") pod \"e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7\" (UID: \"e5944af0-5719-461a-b5bc-0c05023e086d\") " pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:48:31.297602 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.297574    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tspx\" (UniqueName: \"kubernetes.io/projected/e5944af0-5719-461a-b5bc-0c05023e086d-kube-api-access-5tspx\") pod \"e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7\" (UID: \"e5944af0-5719-461a-b5bc-0c05023e086d\") " pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:48:31.298076 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.298056    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:48:31.339920 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.328639    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:31.339920 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.332771    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b913b363-6fbf-4031-a833-2e5e71ef1f19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa138efb-67b2j\" (UID: \"b913b363-6fbf-4031-a833-2e5e71ef1f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa138efb-67b2j"
Oct 30 20:48:31.397379 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.397338    5071 scope.go:122] "RemoveContainer" containerID="83d2bfbe5d1608390995c005672596c1eb2dff11e4c00924166ae63d23ad3af8"
Oct 30 20:48:31.502100 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.467128    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkqqh\" (UniqueName: \"kubernetes.io/projected/b913b363-6fbf-4031-a833-2e5e71ef1f19-kube-api-access-bkqqh\") pod \"maas-api-revoke-sub-aa138efb-67b2j\" (UID: \"b913b363-6fbf-4031-a833-2e5e71ef1f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa138efb-67b2j"
Oct 30 20:48:31.502100 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.467193    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b913b363-6fbf-4031-a833-2e5e71ef1f19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa138efb-67b2j\" (UID: \"b913b363-6fbf-4031-a833-2e5e71ef1f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa138efb-67b2j"
Oct 30 20:48:31.502100 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.467301    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b913b363-6fbf-4031-a833-2e5e71ef1f19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa138efb-67b2j\" (UID: \"b913b363-6fbf-4031-a833-2e5e71ef1f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa138efb-67b2j"
Oct 30 20:48:31.502100 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.494208    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b913b363-6fbf-4031-a833-2e5e71ef1f19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa138efb-67b2j\" (UID: \"b913b363-6fbf-4031-a833-2e5e71ef1f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa138efb-67b2j"
Oct 30 20:48:31.502100 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.495971    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b913b363-6fbf-4031-a833-2e5e71ef1f19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa138efb-67b2j\" (UID: \"b913b363-6fbf-4031-a833-2e5e71ef1f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa138efb-67b2j"
Oct 30 20:48:31.627422 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.627213    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkqqh\" (UniqueName: \"kubernetes.io/projected/b913b363-6fbf-4031-a833-2e5e71ef1f19-kube-api-access-bkqqh\") pod \"maas-api-revoke-sub-aa138efb-67b2j\" (UID: \"b913b363-6fbf-4031-a833-2e5e71ef1f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa138efb-67b2j"
Oct 30 20:48:31.627664 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.627375    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:31.727445 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.725490    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa138efb-67b2j"
Oct 30 20:48:31.758562 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.758471    5071 scope.go:122] "RemoveContainer" containerID="4c89174d9b13969cf4efe062cdb0446d2ce3bc1ba2e669ce557415b0e40ed794"
Oct 30 20:48:31.904645 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.904613    5071 scope.go:122] "RemoveContainer" containerID="f2a0f968594a25a1e594bbcc3a067ad10b1d7bc5069a685bd63ff1cddc8ae17a"
Oct 30 20:48:32.002072 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:31.991236    5071 scope.go:122] "RemoveContainer" containerID="7b62535ac9ae1725ab2e66b73f5f33456f6389d8294a0f850dd96822162fcfa6"
Oct 30 20:48:32.014909 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:32.002511    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"]
Oct 30 20:48:32.091867 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:32.074989    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 20:48:32.091867 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:32.075058    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47" podUID="736322c2-4cdb-4fd1-bbc1-15c089d827d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 20:48:32.092430 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:32.092390    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:48:32.092503 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:32.092485    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:48:32.092559 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:32.092532    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:48:32.103576 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:32.103547    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:32.124593 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:32.124570    5071 scope.go:122] "RemoveContainer" containerID="1b37f262855ab980b0f69a6699dc2da69eabdb11986b08dd68a3b701c5f0247f"
Oct 30 20:48:32.327503 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:48:32.327417    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb913b363_6fbf_4031_a833_2e5e71ef1f19.slice/crio-0dc56c13a04f3dcc889849b268d2b0bdf4bf7c0a95f30fb23332a0a8451d2c3f WatchSource:0}: Error finding container 0dc56c13a04f3dcc889849b268d2b0bdf4bf7c0a95f30fb23332a0a8451d2c3f: Status 404 returned error can't find the container with id 0dc56c13a04f3dcc889849b268d2b0bdf4bf7c0a95f30fb23332a0a8451d2c3f
Oct 30 20:48:32.327840 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:32.327793    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa138efb-67b2j"]
Oct 30 20:48:32.359236 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:32.345264    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:32.476889 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:32.476856    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"]
Oct 30 20:48:32.489934 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:32.489889    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:48:32.490026 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:32.489973    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:48:32.490026 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:32.490017    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:48:32.520364 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:32.520342    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:32.708235 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:32.708039    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:32.923892 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:32.923793    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:32.929857 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:32.926604    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa138efb-67b2j" event={"ID":"b913b363-6fbf-4031-a833-2e5e71ef1f19","Type":"ContainerStarted","Data":"0dc56c13a04f3dcc889849b268d2b0bdf4bf7c0a95f30fb23332a0a8451d2c3f"}
Oct 30 20:48:32.958906 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:32.944416    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c" event={"ID":"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d","Type":"ContainerStarted","Data":"1324533b9d510d2354a3ddc1c62a37116bdc450c04ea33254a87e8877d57ea40"}
Oct 30 20:48:32.958906 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:32.944481    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c" event={"ID":"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d","Type":"ContainerStarted","Data":"acd04669e7a3a30ce8857471c4e0aaa1635dd485de68840bdbed0b6a859198db"}
Oct 30 20:48:32.958906 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:32.955589    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7" event={"ID":"e5944af0-5719-461a-b5bc-0c05023e086d","Type":"ContainerStarted","Data":"33f14c5bd5c7dcf3b1362429cc58fe718aecc5a65d4d44a978c8461a48ed0df5"}
Oct 30 20:48:32.958906 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:32.955650    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7" event={"ID":"e5944af0-5719-461a-b5bc-0c05023e086d","Type":"ContainerStarted","Data":"d1a717db7e7deb0f835d1f82994a9e77a51eb8f8e256fd45f355eb71da3ae3f5"}
Oct 30 20:48:33.053861 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:33.048427    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 20:48:33.053861 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:33.048479    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47" podUID="736322c2-4cdb-4fd1-bbc1-15c089d827d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 20:48:33.075125 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:33.075047    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:48:33.095251 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:33.095214    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:48:33.124339 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:33.124289    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c" podStartSLOduration=7.124275037 podStartE2EDuration="7.124275037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:33.119955997 +0000 UTC m=+1325.036977545" watchObservedRunningTime="2026-10-30 20:48:33.124275037 +0000 UTC m=+1325.041296573"
Oct 30 20:48:33.285964 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:33.285781    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7" podStartSLOduration=4.28575804 podStartE2EDuration="4.28575804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:33.244734432 +0000 UTC m=+1325.161755969" watchObservedRunningTime="2026-10-30 20:48:33.28575804 +0000 UTC m=+1325.202779576"
Oct 30 20:48:33.313847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:33.313485    5071 patch_prober.go:29] interesting pod/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 20:48:33.313847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:33.313548    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7" podUID="e5944af0-5719-461a-b5bc-0c05023e086d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 20:48:33.314183 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:33.314123    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:48:33.691285 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:48:33.691246    5071 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb913b363_6fbf_4031_a833_2e5e71ef1f19.slice/crio-09cda05ac8d3877a10953680fa99880359ce54e4c0486e41433684be7ab300b0.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:48:33.971130 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:33.971022    5071 generic.go:356] "Generic (PLEG): container finished" podID="b913b363-6fbf-4031-a833-2e5e71ef1f19" containerID="09cda05ac8d3877a10953680fa99880359ce54e4c0486e41433684be7ab300b0" exitCode=0
Oct 30 20:48:33.971130 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:33.971109    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa138efb-67b2j" event={"ID":"b913b363-6fbf-4031-a833-2e5e71ef1f19","Type":"ContainerDied","Data":"09cda05ac8d3877a10953680fa99880359ce54e4c0486e41433684be7ab300b0"}
Oct 30 20:48:33.971745 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:33.971723    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:48:33.972689 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:33.972669    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:48:34.019418 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:34.019382    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 20:48:34.019628 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:34.019437    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47" podUID="736322c2-4cdb-4fd1-bbc1-15c089d827d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 20:48:34.310875 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:34.310810    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:48:34.982324 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:34.982289    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:48:34.984230 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:34.984201    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:48:35.192090 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:35.192059    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:35.215813 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:35.215783    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa138efb-67b2j"
Oct 30 20:48:35.299990 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:35.276475    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b913b363-6fbf-4031-a833-2e5e71ef1f19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b913b363-6fbf-4031-a833-2e5e71ef1f19-maas-api-openshift-ca\") pod \"b913b363-6fbf-4031-a833-2e5e71ef1f19\" (UID: \"b913b363-6fbf-4031-a833-2e5e71ef1f19\") "
Oct 30 20:48:35.300395 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:35.300358    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b913b363-6fbf-4031-a833-2e5e71ef1f19-kube-api-access-bkqqh\" (UniqueName: \"kubernetes.io/projected/b913b363-6fbf-4031-a833-2e5e71ef1f19-kube-api-access-bkqqh\") pod \"b913b363-6fbf-4031-a833-2e5e71ef1f19\" (UID: \"b913b363-6fbf-4031-a833-2e5e71ef1f19\") "
Oct 30 20:48:35.318333 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:35.318297    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b913b363-6fbf-4031-a833-2e5e71ef1f19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b913b363-6fbf-4031-a833-2e5e71ef1f19-maas-api-xks-ca\") pod \"b913b363-6fbf-4031-a833-2e5e71ef1f19\" (UID: \"b913b363-6fbf-4031-a833-2e5e71ef1f19\") "
Oct 30 20:48:35.322400 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:35.282696    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b913b363-6fbf-4031-a833-2e5e71ef1f19-maas-api-openshift-ca" pod "b913b363-6fbf-4031-a833-2e5e71ef1f19" (UID: "b913b363-6fbf-4031-a833-2e5e71ef1f19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:48:35.325769 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:35.325723    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b913b363-6fbf-4031-a833-2e5e71ef1f19-maas-api-xks-ca" pod "b913b363-6fbf-4031-a833-2e5e71ef1f19" (UID: "b913b363-6fbf-4031-a833-2e5e71ef1f19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:48:35.380180 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:35.380147    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"
Oct 30 20:48:35.381972 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:35.381880    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:35.557717 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:35.557416    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"
Oct 30 20:48:35.560707 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:35.559361    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b913b363-6fbf-4031-a833-2e5e71ef1f19-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:35.560707 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:35.559389    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b913b363-6fbf-4031-a833-2e5e71ef1f19-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:35.628014 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:35.627979    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b913b363-6fbf-4031-a833-2e5e71ef1f19-kube-api-access-bkqqh" pod "b913b363-6fbf-4031-a833-2e5e71ef1f19" (UID: "b913b363-6fbf-4031-a833-2e5e71ef1f19"). InnerVolumeSpecName "kube-api-access-bkqqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:48:35.661451 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:35.661390    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkqqh\" (UniqueName: \"kubernetes.io/projected/b913b363-6fbf-4031-a833-2e5e71ef1f19-kube-api-access-bkqqh\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:36.044181 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:35.963406    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-dc0e9f64-66454765d-kn2vl"]
Oct 30 20:48:36.044181 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:35.964581    5071 cpu_manager.go:414] "containerMap: removing container" podUID="b913b363-6fbf-4031-a833-2e5e71ef1f19" containerName="revoke-keys"
Oct 30 20:48:36.044181 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:35.964604    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b913b363-6fbf-4031-a833-2e5e71ef1f19" containerName="revoke-keys"
Oct 30 20:48:36.044181 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:35.964871    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="b913b363-6fbf-4031-a833-2e5e71ef1f19" containerName="revoke-keys"
Oct 30 20:48:36.173877 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:36.165232    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-dc0e9f64-66454765d-kn2vl"
Oct 30 20:48:36.218815 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:36.218780    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa138efb-67b2j"
Oct 30 20:48:36.248212 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:36.243569    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-dc0e9f64\""
Oct 30 20:48:36.248212 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:36.244278    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-dc0e9f64-dockercfg-fcgsv\""
Oct 30 20:48:36.273598 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:36.273571    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-dc0e9f64-66454765d-kn2vl"]
Oct 30 20:48:36.273744 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:36.273626    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa138efb-67b2j" event={"ID":"b913b363-6fbf-4031-a833-2e5e71ef1f19","Type":"ContainerDied","Data":"0dc56c13a04f3dcc889849b268d2b0bdf4bf7c0a95f30fb23332a0a8451d2c3f"}
Oct 30 20:48:36.273744 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:36.273649    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0dc56c13a04f3dcc889849b268d2b0bdf4bf7c0a95f30fb23332a0a8451d2c3f"
Oct 30 20:48:36.321472 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:36.321393    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:36.348734 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:36.348694    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aac6152c-a8ea-4988-996a-e132f9c29c18-plugins-config-volume\") pod \"payload-processing-e2e-migrate-dc0e9f64-66454765d-kn2vl\" (UID: \"aac6152c-a8ea-4988-996a-e132f9c29c18\") " pod="openshift-ingress/payload-processing-e2e-migrate-dc0e9f64-66454765d-kn2vl"
Oct 30 20:48:36.349312 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:36.349283    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27rgp\" (UniqueName: \"kubernetes.io/projected/aac6152c-a8ea-4988-996a-e132f9c29c18-kube-api-access-27rgp\") pod \"payload-processing-e2e-migrate-dc0e9f64-66454765d-kn2vl\" (UID: \"aac6152c-a8ea-4988-996a-e132f9c29c18\") " pod="openshift-ingress/payload-processing-e2e-migrate-dc0e9f64-66454765d-kn2vl"
Oct 30 20:48:36.451472 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:36.451422    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aac6152c-a8ea-4988-996a-e132f9c29c18-plugins-config-volume\") pod \"payload-processing-e2e-migrate-dc0e9f64-66454765d-kn2vl\" (UID: \"aac6152c-a8ea-4988-996a-e132f9c29c18\") " pod="openshift-ingress/payload-processing-e2e-migrate-dc0e9f64-66454765d-kn2vl"
Oct 30 20:48:36.453791 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:36.452644    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aac6152c-a8ea-4988-996a-e132f9c29c18-plugins-config-volume\") pod \"payload-processing-e2e-migrate-dc0e9f64-66454765d-kn2vl\" (UID: \"aac6152c-a8ea-4988-996a-e132f9c29c18\") " pod="openshift-ingress/payload-processing-e2e-migrate-dc0e9f64-66454765d-kn2vl"
Oct 30 20:48:36.454133 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:36.454110    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27rgp\" (UniqueName: \"kubernetes.io/projected/aac6152c-a8ea-4988-996a-e132f9c29c18-kube-api-access-27rgp\") pod \"payload-processing-e2e-migrate-dc0e9f64-66454765d-kn2vl\" (UID: \"aac6152c-a8ea-4988-996a-e132f9c29c18\") " pod="openshift-ingress/payload-processing-e2e-migrate-dc0e9f64-66454765d-kn2vl"
Oct 30 20:48:36.516960 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:36.516929    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27rgp\" (UniqueName: \"kubernetes.io/projected/aac6152c-a8ea-4988-996a-e132f9c29c18-kube-api-access-27rgp\") pod \"payload-processing-e2e-migrate-dc0e9f64-66454765d-kn2vl\" (UID: \"aac6152c-a8ea-4988-996a-e132f9c29c18\") " pod="openshift-ingress/payload-processing-e2e-migrate-dc0e9f64-66454765d-kn2vl"
Oct 30 20:48:36.541845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:36.538980    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:36.565845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:36.565169    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-dc0e9f64-66454765d-kn2vl"
Oct 30 20:48:36.710627 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:36.710532    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:36.991663 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:36.991569    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-dc0e9f64-66454765d-kn2vl"]
Oct 30 20:48:36.998925 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:48:36.998895    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaac6152c_a8ea_4988_996a_e132f9c29c18.slice/crio-adae78ae8fdc1668422c61df7071ed5b8f6083a0e5cf8aada8f4d56e8c5670f7 WatchSource:0}: Error finding container adae78ae8fdc1668422c61df7071ed5b8f6083a0e5cf8aada8f4d56e8c5670f7: Status 404 returned error can't find the container with id adae78ae8fdc1668422c61df7071ed5b8f6083a0e5cf8aada8f4d56e8c5670f7
Oct 30 20:48:37.046659 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:37.046621    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:37.234713 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:37.234676    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-dc0e9f64-66454765d-kn2vl" event={"ID":"aac6152c-a8ea-4988-996a-e132f9c29c18","Type":"ContainerStarted","Data":"bac6abe423f3aba15db82c8f82eed035a79508ad42827e1a54eca96b5723cce6"}
Oct 30 20:48:37.234922 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:37.234726    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-dc0e9f64-66454765d-kn2vl" event={"ID":"aac6152c-a8ea-4988-996a-e132f9c29c18","Type":"ContainerStarted","Data":"adae78ae8fdc1668422c61df7071ed5b8f6083a0e5cf8aada8f4d56e8c5670f7"}
Oct 30 20:48:37.234922 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:37.234759    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-dc0e9f64-66454765d-kn2vl"
Oct 30 20:48:37.343770 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:37.343734    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:37.364894 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:37.364843    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-dc0e9f64-66454765d-kn2vl" podStartSLOduration=2.364811162 podStartE2EDuration="2.364811162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:37.358419004 +0000 UTC m=+1329.275440540" watchObservedRunningTime="2026-10-30 20:48:37.364811162 +0000 UTC m=+1329.281832698"
Oct 30 20:48:37.523521 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:37.523477    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:37.714388 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:37.714303    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:37.914146 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:37.914111    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:38.093847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:38.092798    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:40.290742 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:40.290692    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:40.528474 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:40.528430    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:42.572379 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:42.572343    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:42.780277 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:42.780232    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:42.984162 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:42.984058    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:43.306032 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:43.280408    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:43.496848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:43.481369    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-78ca07a3-db58645f-ckkt9"]
Oct 30 20:48:43.779982 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:43.773428    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:43.948847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:43.946319    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-78ca07a3-db58645f-ckkt9"]
Oct 30 20:48:43.948847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:43.946470    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-78ca07a3-db58645f-ckkt9"
Oct 30 20:48:43.960990 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:43.960896    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-78ca07a3-6f8759c9f8-xz5m9"]
Oct 30 20:48:43.970345 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:43.967253    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-78ca07a3-6f8759c9f8-xz5m9"]
Oct 30 20:48:43.970345 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:43.967379    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-78ca07a3-6f8759c9f8-xz5m9"
Oct 30 20:48:43.997239 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:43.997208    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtw2h\" (UniqueName: \"kubernetes.io/projected/7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d-kube-api-access-jtw2h\") pod \"payload-pre-processing-e2e-ait-78ca07a3-db58645f-ckkt9\" (UID: \"7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-78ca07a3-db58645f-ckkt9"
Oct 30 20:48:43.997457 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:43.997436    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-78ca07a3-db58645f-ckkt9\" (UID: \"7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-78ca07a3-db58645f-ckkt9"
Oct 30 20:48:44.001564 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:44.001546    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-78ca07a3\""
Oct 30 20:48:44.050617 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:44.050593    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-78ca07a3-dockercfg-t6v6k\""
Oct 30 20:48:44.098592 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:44.098557    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph846\" (UniqueName: \"kubernetes.io/projected/52c6baf2-74c7-4ee6-baee-7357190687e5-kube-api-access-ph846\") pod \"payload-processing-e2e-ait-78ca07a3-6f8759c9f8-xz5m9\" (UID: \"52c6baf2-74c7-4ee6-baee-7357190687e5\") " pod="openshift-ingress/payload-processing-e2e-ait-78ca07a3-6f8759c9f8-xz5m9"
Oct 30 20:48:44.098745 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:44.098603    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtw2h\" (UniqueName: \"kubernetes.io/projected/7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d-kube-api-access-jtw2h\") pod \"payload-pre-processing-e2e-ait-78ca07a3-db58645f-ckkt9\" (UID: \"7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-78ca07a3-db58645f-ckkt9"
Oct 30 20:48:44.098870 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:44.098842    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52c6baf2-74c7-4ee6-baee-7357190687e5-plugins-config-volume\") pod \"payload-processing-e2e-ait-78ca07a3-6f8759c9f8-xz5m9\" (UID: \"52c6baf2-74c7-4ee6-baee-7357190687e5\") " pod="openshift-ingress/payload-processing-e2e-ait-78ca07a3-6f8759c9f8-xz5m9"
Oct 30 20:48:44.098910 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:44.098891    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-78ca07a3-db58645f-ckkt9\" (UID: \"7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-78ca07a3-db58645f-ckkt9"
Oct 30 20:48:44.099734 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:44.099715    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-78ca07a3-db58645f-ckkt9\" (UID: \"7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-78ca07a3-db58645f-ckkt9"
Oct 30 20:48:44.158090 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:44.158061    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtw2h\" (UniqueName: \"kubernetes.io/projected/7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d-kube-api-access-jtw2h\") pod \"payload-pre-processing-e2e-ait-78ca07a3-db58645f-ckkt9\" (UID: \"7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-78ca07a3-db58645f-ckkt9"
Oct 30 20:48:44.200521 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:44.200492    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph846\" (UniqueName: \"kubernetes.io/projected/52c6baf2-74c7-4ee6-baee-7357190687e5-kube-api-access-ph846\") pod \"payload-processing-e2e-ait-78ca07a3-6f8759c9f8-xz5m9\" (UID: \"52c6baf2-74c7-4ee6-baee-7357190687e5\") " pod="openshift-ingress/payload-processing-e2e-ait-78ca07a3-6f8759c9f8-xz5m9"
Oct 30 20:48:44.200707 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:44.200684    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52c6baf2-74c7-4ee6-baee-7357190687e5-plugins-config-volume\") pod \"payload-processing-e2e-ait-78ca07a3-6f8759c9f8-xz5m9\" (UID: \"52c6baf2-74c7-4ee6-baee-7357190687e5\") " pod="openshift-ingress/payload-processing-e2e-ait-78ca07a3-6f8759c9f8-xz5m9"
Oct 30 20:48:44.201628 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:44.201606    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52c6baf2-74c7-4ee6-baee-7357190687e5-plugins-config-volume\") pod \"payload-processing-e2e-ait-78ca07a3-6f8759c9f8-xz5m9\" (UID: \"52c6baf2-74c7-4ee6-baee-7357190687e5\") " pod="openshift-ingress/payload-processing-e2e-ait-78ca07a3-6f8759c9f8-xz5m9"
Oct 30 20:48:44.258756 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:44.258732    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph846\" (UniqueName: \"kubernetes.io/projected/52c6baf2-74c7-4ee6-baee-7357190687e5-kube-api-access-ph846\") pod \"payload-processing-e2e-ait-78ca07a3-6f8759c9f8-xz5m9\" (UID: \"52c6baf2-74c7-4ee6-baee-7357190687e5\") " pod="openshift-ingress/payload-processing-e2e-ait-78ca07a3-6f8759c9f8-xz5m9"
Oct 30 20:48:44.275941 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:44.275914    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-78ca07a3-db58645f-ckkt9"
Oct 30 20:48:44.286891 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:44.286863    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-78ca07a3-6f8759c9f8-xz5m9"
Oct 30 20:48:44.764942 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:48:44.764908    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a3de0a4_ec0f_4f29_b4bc_c5858e95e73d.slice/crio-984e50dda07437b86d815185fcf04c04b02a45e21f37f19203ac6a16388fffaf WatchSource:0}: Error finding container 984e50dda07437b86d815185fcf04c04b02a45e21f37f19203ac6a16388fffaf: Status 404 returned error can't find the container with id 984e50dda07437b86d815185fcf04c04b02a45e21f37f19203ac6a16388fffaf
Oct 30 20:48:44.767461 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:44.767429    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-78ca07a3-db58645f-ckkt9"]
Oct 30 20:48:44.777798 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:44.777768    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-78ca07a3-6f8759c9f8-xz5m9"]
Oct 30 20:48:44.809393 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:48:44.809362    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c6baf2_74c7_4ee6_baee_7357190687e5.slice/crio-645813d0dcd401b5fe0d0fa7671dc74dfd6bc8cd804549ed2009ad47431ad986 WatchSource:0}: Error finding container 645813d0dcd401b5fe0d0fa7671dc74dfd6bc8cd804549ed2009ad47431ad986: Status 404 returned error can't find the container with id 645813d0dcd401b5fe0d0fa7671dc74dfd6bc8cd804549ed2009ad47431ad986
Oct 30 20:48:44.869019 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:44.868988    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-78ca07a3-6f8759c9f8-xz5m9" event={"ID":"52c6baf2-74c7-4ee6-baee-7357190687e5","Type":"ContainerStarted","Data":"645813d0dcd401b5fe0d0fa7671dc74dfd6bc8cd804549ed2009ad47431ad986"}
Oct 30 20:48:44.870962 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:44.870917    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-78ca07a3-db58645f-ckkt9" event={"ID":"7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d","Type":"ContainerStarted","Data":"a407650503221dcf29bdc6c94eeb0364cffb72adffccf45ada787c9f8c922a15"}
Oct 30 20:48:44.871069 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:44.870979    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-78ca07a3-db58645f-ckkt9" event={"ID":"7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d","Type":"ContainerStarted","Data":"984e50dda07437b86d815185fcf04c04b02a45e21f37f19203ac6a16388fffaf"}
Oct 30 20:48:44.988654 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:44.988592    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-78ca07a3-db58645f-ckkt9" podStartSLOduration=1.988579313 podStartE2EDuration="1.988579313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:44.985349892 +0000 UTC m=+1336.902371429" watchObservedRunningTime="2026-10-30 20:48:44.988579313 +0000 UTC m=+1336.905600849"
Oct 30 20:48:45.596461 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:45.596427    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:45.954043 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:45.953952    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"]
Oct 30 20:48:45.955804 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:45.955774    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:45.959561 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:45.959528    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-78ca07a3-6f8759c9f8-xz5m9" event={"ID":"52c6baf2-74c7-4ee6-baee-7357190687e5","Type":"ContainerStarted","Data":"c9e1dcd5aacfcd07d0383f0cbad36ba31431d232235f8c0b5e40758cb81f59e5"}
Oct 30 20:48:45.959675 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:45.959575    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-78ca07a3-db58645f-ckkt9"
Oct 30 20:48:45.959675 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:45.959636    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"
Oct 30 20:48:45.960134 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:45.960111    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-78ca07a3-6f8759c9f8-xz5m9"
Oct 30 20:48:45.999621 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:45.999589    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bd92310f-d9b4b64d7-pbvp7"]
Oct 30 20:48:46.004699 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.004672    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bd92310f-d9b4b64d7-pbvp7"
Oct 30 20:48:46.061717 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.061688    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-bd92310f\""
Oct 30 20:48:46.062216 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.062185    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-bd92310f-dockercfg-bkj62\""
Oct 30 20:48:46.115843 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.115579    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bd92310f-d9b4b64d7-pbvp7"]
Oct 30 20:48:46.150856 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.148069    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/576450be-aec9-44a7-a91b-b2bd82342e57-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-bd92310f-d9b4b64d7-pbvp7\" (UID: \"576450be-aec9-44a7-a91b-b2bd82342e57\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bd92310f-d9b4b64d7-pbvp7"
Oct 30 20:48:46.150856 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.148132    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b52815-7184-4c7b-a294-a2d9a7d0c099-workload-certs\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92\" (UID: \"31b52815-7184-4c7b-a294-a2d9a7d0c099\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"
Oct 30 20:48:46.150856 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.148242    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b52815-7184-4c7b-a294-a2d9a7d0c099-credential-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92\" (UID: \"31b52815-7184-4c7b-a294-a2d9a7d0c099\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"
Oct 30 20:48:46.150856 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.148309    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b52815-7184-4c7b-a294-a2d9a7d0c099-istio-envoy\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92\" (UID: \"31b52815-7184-4c7b-a294-a2d9a7d0c099\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"
Oct 30 20:48:46.150856 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.148369    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-954n4\" (UniqueName: \"kubernetes.io/projected/576450be-aec9-44a7-a91b-b2bd82342e57-kube-api-access-954n4\") pod \"payload-pre-processing-e2e-derive-bd92310f-d9b4b64d7-pbvp7\" (UID: \"576450be-aec9-44a7-a91b-b2bd82342e57\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bd92310f-d9b4b64d7-pbvp7"
Oct 30 20:48:46.150856 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.148419    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b52815-7184-4c7b-a294-a2d9a7d0c099-istio-data\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92\" (UID: \"31b52815-7184-4c7b-a294-a2d9a7d0c099\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"
Oct 30 20:48:46.150856 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.148459    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b52815-7184-4c7b-a294-a2d9a7d0c099-istio-token\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92\" (UID: \"31b52815-7184-4c7b-a294-a2d9a7d0c099\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"
Oct 30 20:48:46.150856 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.148520    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b52815-7184-4c7b-a294-a2d9a7d0c099-workload-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92\" (UID: \"31b52815-7184-4c7b-a294-a2d9a7d0c099\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"
Oct 30 20:48:46.150856 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.148571    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b52815-7184-4c7b-a294-a2d9a7d0c099-istio-podinfo\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92\" (UID: \"31b52815-7184-4c7b-a294-a2d9a7d0c099\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"
Oct 30 20:48:46.150856 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.148630    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wzgt\" (UniqueName: \"kubernetes.io/projected/31b52815-7184-4c7b-a294-a2d9a7d0c099-kube-api-access-4wzgt\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92\" (UID: \"31b52815-7184-4c7b-a294-a2d9a7d0c099\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"
Oct 30 20:48:46.150856 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.148711    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b52815-7184-4c7b-a294-a2d9a7d0c099-istiod-ca-cert\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92\" (UID: \"31b52815-7184-4c7b-a294-a2d9a7d0c099\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"
Oct 30 20:48:46.150856 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.148862    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-78ca07a3-6f8759c9f8-xz5m9" podStartSLOduration=3.148841707 podStartE2EDuration="3.148841707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:46.120278948 +0000 UTC m=+1338.037300484" watchObservedRunningTime="2026-10-30 20:48:46.148841707 +0000 UTC m=+1338.065863244"
Oct 30 20:48:46.254903 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.254405    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/576450be-aec9-44a7-a91b-b2bd82342e57-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-bd92310f-d9b4b64d7-pbvp7\" (UID: \"576450be-aec9-44a7-a91b-b2bd82342e57\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bd92310f-d9b4b64d7-pbvp7"
Oct 30 20:48:46.254903 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.254462    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b52815-7184-4c7b-a294-a2d9a7d0c099-workload-certs\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92\" (UID: \"31b52815-7184-4c7b-a294-a2d9a7d0c099\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"
Oct 30 20:48:46.254903 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.254567    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b52815-7184-4c7b-a294-a2d9a7d0c099-credential-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92\" (UID: \"31b52815-7184-4c7b-a294-a2d9a7d0c099\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"
Oct 30 20:48:46.254903 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.254627    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b52815-7184-4c7b-a294-a2d9a7d0c099-istio-envoy\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92\" (UID: \"31b52815-7184-4c7b-a294-a2d9a7d0c099\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"
Oct 30 20:48:46.254903 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.254691    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-954n4\" (UniqueName: \"kubernetes.io/projected/576450be-aec9-44a7-a91b-b2bd82342e57-kube-api-access-954n4\") pod \"payload-pre-processing-e2e-derive-bd92310f-d9b4b64d7-pbvp7\" (UID: \"576450be-aec9-44a7-a91b-b2bd82342e57\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bd92310f-d9b4b64d7-pbvp7"
Oct 30 20:48:46.254903 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.254733    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b52815-7184-4c7b-a294-a2d9a7d0c099-istio-data\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92\" (UID: \"31b52815-7184-4c7b-a294-a2d9a7d0c099\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"
Oct 30 20:48:46.254903 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.254772    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b52815-7184-4c7b-a294-a2d9a7d0c099-istio-token\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92\" (UID: \"31b52815-7184-4c7b-a294-a2d9a7d0c099\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"
Oct 30 20:48:46.254903 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.254849    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b52815-7184-4c7b-a294-a2d9a7d0c099-workload-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92\" (UID: \"31b52815-7184-4c7b-a294-a2d9a7d0c099\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"
Oct 30 20:48:46.254903 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.254902    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b52815-7184-4c7b-a294-a2d9a7d0c099-istio-podinfo\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92\" (UID: \"31b52815-7184-4c7b-a294-a2d9a7d0c099\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"
Oct 30 20:48:46.281870 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.254958    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wzgt\" (UniqueName: \"kubernetes.io/projected/31b52815-7184-4c7b-a294-a2d9a7d0c099-kube-api-access-4wzgt\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92\" (UID: \"31b52815-7184-4c7b-a294-a2d9a7d0c099\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"
Oct 30 20:48:46.281870 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.255031    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b52815-7184-4c7b-a294-a2d9a7d0c099-istiod-ca-cert\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92\" (UID: \"31b52815-7184-4c7b-a294-a2d9a7d0c099\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"
Oct 30 20:48:46.281870 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.256008    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b52815-7184-4c7b-a294-a2d9a7d0c099-istiod-ca-cert\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92\" (UID: \"31b52815-7184-4c7b-a294-a2d9a7d0c099\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"
Oct 30 20:48:46.282160 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.282058    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b52815-7184-4c7b-a294-a2d9a7d0c099-istio-data\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92\" (UID: \"31b52815-7184-4c7b-a294-a2d9a7d0c099\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"
Oct 30 20:48:46.305854 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.282305    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b52815-7184-4c7b-a294-a2d9a7d0c099-workload-certs\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92\" (UID: \"31b52815-7184-4c7b-a294-a2d9a7d0c099\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"
Oct 30 20:48:46.305854 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.282543    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b52815-7184-4c7b-a294-a2d9a7d0c099-credential-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92\" (UID: \"31b52815-7184-4c7b-a294-a2d9a7d0c099\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"
Oct 30 20:48:46.322526 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.322483    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b52815-7184-4c7b-a294-a2d9a7d0c099-workload-socket\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92\" (UID: \"31b52815-7184-4c7b-a294-a2d9a7d0c099\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"
Oct 30 20:48:46.360103 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.360052    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/576450be-aec9-44a7-a91b-b2bd82342e57-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-bd92310f-d9b4b64d7-pbvp7\" (UID: \"576450be-aec9-44a7-a91b-b2bd82342e57\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bd92310f-d9b4b64d7-pbvp7"
Oct 30 20:48:46.371861 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.370476    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b52815-7184-4c7b-a294-a2d9a7d0c099-istio-envoy\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92\" (UID: \"31b52815-7184-4c7b-a294-a2d9a7d0c099\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"
Oct 30 20:48:46.453604 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.453559    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b52815-7184-4c7b-a294-a2d9a7d0c099-istio-podinfo\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92\" (UID: \"31b52815-7184-4c7b-a294-a2d9a7d0c099\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"
Oct 30 20:48:46.473926 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.472163    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-954n4\" (UniqueName: \"kubernetes.io/projected/576450be-aec9-44a7-a91b-b2bd82342e57-kube-api-access-954n4\") pod \"payload-pre-processing-e2e-derive-bd92310f-d9b4b64d7-pbvp7\" (UID: \"576450be-aec9-44a7-a91b-b2bd82342e57\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bd92310f-d9b4b64d7-pbvp7"
Oct 30 20:48:46.481403 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.481319    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"]
Oct 30 20:48:46.485503 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.485438    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b52815-7184-4c7b-a294-a2d9a7d0c099-istio-token\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92\" (UID: \"31b52815-7184-4c7b-a294-a2d9a7d0c099\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"
Oct 30 20:48:46.493870 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.491115    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wzgt\" (UniqueName: \"kubernetes.io/projected/31b52815-7184-4c7b-a294-a2d9a7d0c099-kube-api-access-4wzgt\") pod \"e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92\" (UID: \"31b52815-7184-4c7b-a294-a2d9a7d0c099\") " pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"
Oct 30 20:48:46.568455 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.561308    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:46.575537 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.575488    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bd92310f-7f56b44dcd-txxfn"]
Oct 30 20:48:46.583198 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.583169    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"
Oct 30 20:48:46.687747 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.687706    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bd92310f-d9b4b64d7-pbvp7"
Oct 30 20:48:46.697297 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.696645    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-bd92310f-7f56b44dcd-txxfn"
Oct 30 20:48:46.700980 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.700580    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bd92310f-7f56b44dcd-txxfn"]
Oct 30 20:48:46.715966 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.715944    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"]
Oct 30 20:48:46.727898 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.727871    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"
Oct 30 20:48:46.736223 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.736193    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"]
Oct 30 20:48:46.744864 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.744764    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"]
Oct 30 20:48:46.745004 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.744907    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"
Oct 30 20:48:46.747815 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:48:46.747113    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:48:46.748280 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.748260    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"]
Oct 30 20:48:46.763195 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.755454    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"]
Oct 30 20:48:46.763461 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.763445    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"]
Oct 30 20:48:46.763610 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.763600    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"
Oct 30 20:48:46.902016 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.901984    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd81aa48-0f88-44fb-8baa-db1deec4a2df-workload-certs\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s\" (UID: \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"
Oct 30 20:48:46.902197 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.902037    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa5037cf-aea2-403e-9865-8df55fe7e04f-workload-certs\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl\" (UID: \"aa5037cf-aea2-403e-9865-8df55fe7e04f\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"
Oct 30 20:48:46.902197 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.902115    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scxx5\" (UniqueName: \"kubernetes.io/projected/eae68bfa-6442-4208-b363-71f7953d875e-kube-api-access-scxx5\") pod \"payload-processing-e2e-derive-bd92310f-7f56b44dcd-txxfn\" (UID: \"eae68bfa-6442-4208-b363-71f7953d875e\") " pod="openshift-ingress/payload-processing-e2e-derive-bd92310f-7f56b44dcd-txxfn"
Oct 30 20:48:46.902197 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.902151    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/903e1c2f-42e8-4702-8368-3831f717fd8e-istio-envoy\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-prccm\" (UID: \"903e1c2f-42e8-4702-8368-3831f717fd8e\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"
Oct 30 20:48:46.902197 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.902176    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/903e1c2f-42e8-4702-8368-3831f717fd8e-istio-token\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-prccm\" (UID: \"903e1c2f-42e8-4702-8368-3831f717fd8e\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"
Oct 30 20:48:46.902452 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.902247    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd81aa48-0f88-44fb-8baa-db1deec4a2df-istio-podinfo\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s\" (UID: \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"
Oct 30 20:48:46.902452 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.902359    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6jjt\" (UniqueName: \"kubernetes.io/projected/903e1c2f-42e8-4702-8368-3831f717fd8e-kube-api-access-t6jjt\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-prccm\" (UID: \"903e1c2f-42e8-4702-8368-3831f717fd8e\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"
Oct 30 20:48:46.902452 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.902390    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/903e1c2f-42e8-4702-8368-3831f717fd8e-credential-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-prccm\" (UID: \"903e1c2f-42e8-4702-8368-3831f717fd8e\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"
Oct 30 20:48:46.902618 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.902462    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa5037cf-aea2-403e-9865-8df55fe7e04f-istiod-ca-cert\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl\" (UID: \"aa5037cf-aea2-403e-9865-8df55fe7e04f\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"
Oct 30 20:48:46.902618 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.902492    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd81aa48-0f88-44fb-8baa-db1deec4a2df-istio-envoy\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s\" (UID: \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"
Oct 30 20:48:46.902618 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.902526    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd81aa48-0f88-44fb-8baa-db1deec4a2df-istio-token\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s\" (UID: \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"
Oct 30 20:48:46.902618 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.902550    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa5037cf-aea2-403e-9865-8df55fe7e04f-istio-data\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl\" (UID: \"aa5037cf-aea2-403e-9865-8df55fe7e04f\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"
Oct 30 20:48:46.902618 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.902601    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd81aa48-0f88-44fb-8baa-db1deec4a2df-workload-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s\" (UID: \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"
Oct 30 20:48:46.902940 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.902626    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa5037cf-aea2-403e-9865-8df55fe7e04f-istio-token\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl\" (UID: \"aa5037cf-aea2-403e-9865-8df55fe7e04f\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"
Oct 30 20:48:46.902940 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.902671    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa5037cf-aea2-403e-9865-8df55fe7e04f-istio-podinfo\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl\" (UID: \"aa5037cf-aea2-403e-9865-8df55fe7e04f\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"
Oct 30 20:48:46.902940 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.902743    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd81aa48-0f88-44fb-8baa-db1deec4a2df-credential-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s\" (UID: \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"
Oct 30 20:48:46.902940 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.902765    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd81aa48-0f88-44fb-8baa-db1deec4a2df-istio-data\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s\" (UID: \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"
Oct 30 20:48:46.902940 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.902813    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eae68bfa-6442-4208-b363-71f7953d875e-plugins-config-volume\") pod \"payload-processing-e2e-derive-bd92310f-7f56b44dcd-txxfn\" (UID: \"eae68bfa-6442-4208-b363-71f7953d875e\") " pod="openshift-ingress/payload-processing-e2e-derive-bd92310f-7f56b44dcd-txxfn"
Oct 30 20:48:46.902940 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.902875    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjxq5\" (UniqueName: \"kubernetes.io/projected/fd81aa48-0f88-44fb-8baa-db1deec4a2df-kube-api-access-hjxq5\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s\" (UID: \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"
Oct 30 20:48:46.902940 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.902908    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5037cf-aea2-403e-9865-8df55fe7e04f-credential-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl\" (UID: \"aa5037cf-aea2-403e-9865-8df55fe7e04f\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"
Oct 30 20:48:46.903308 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.902971    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa5037cf-aea2-403e-9865-8df55fe7e04f-istio-envoy\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl\" (UID: \"aa5037cf-aea2-403e-9865-8df55fe7e04f\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"
Oct 30 20:48:46.903308 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.902998    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/903e1c2f-42e8-4702-8368-3831f717fd8e-workload-certs\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-prccm\" (UID: \"903e1c2f-42e8-4702-8368-3831f717fd8e\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"
Oct 30 20:48:46.903308 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.903021    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/903e1c2f-42e8-4702-8368-3831f717fd8e-istio-podinfo\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-prccm\" (UID: \"903e1c2f-42e8-4702-8368-3831f717fd8e\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"
Oct 30 20:48:46.903308 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.903125    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lczf9\" (UniqueName: \"kubernetes.io/projected/aa5037cf-aea2-403e-9865-8df55fe7e04f-kube-api-access-lczf9\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl\" (UID: \"aa5037cf-aea2-403e-9865-8df55fe7e04f\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"
Oct 30 20:48:46.903308 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.903152    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5037cf-aea2-403e-9865-8df55fe7e04f-workload-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl\" (UID: \"aa5037cf-aea2-403e-9865-8df55fe7e04f\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"
Oct 30 20:48:46.903574 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.903305    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/903e1c2f-42e8-4702-8368-3831f717fd8e-istio-data\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-prccm\" (UID: \"903e1c2f-42e8-4702-8368-3831f717fd8e\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"
Oct 30 20:48:46.903574 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.903341    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/903e1c2f-42e8-4702-8368-3831f717fd8e-istiod-ca-cert\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-prccm\" (UID: \"903e1c2f-42e8-4702-8368-3831f717fd8e\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"
Oct 30 20:48:46.903574 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.903380    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd81aa48-0f88-44fb-8baa-db1deec4a2df-istiod-ca-cert\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s\" (UID: \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"
Oct 30 20:48:46.903574 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:46.903470    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/903e1c2f-42e8-4702-8368-3831f717fd8e-workload-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-prccm\" (UID: \"903e1c2f-42e8-4702-8368-3831f717fd8e\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"
Oct 30 20:48:47.005034 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.004995    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5037cf-aea2-403e-9865-8df55fe7e04f-workload-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl\" (UID: \"aa5037cf-aea2-403e-9865-8df55fe7e04f\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"
Oct 30 20:48:47.005458 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.005044    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/903e1c2f-42e8-4702-8368-3831f717fd8e-istio-data\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-prccm\" (UID: \"903e1c2f-42e8-4702-8368-3831f717fd8e\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"
Oct 30 20:48:47.005458 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.005061    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/903e1c2f-42e8-4702-8368-3831f717fd8e-istiod-ca-cert\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-prccm\" (UID: \"903e1c2f-42e8-4702-8368-3831f717fd8e\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"
Oct 30 20:48:47.005458 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.005159    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd81aa48-0f88-44fb-8baa-db1deec4a2df-istiod-ca-cert\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s\" (UID: \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"
Oct 30 20:48:47.005458 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.005193    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/903e1c2f-42e8-4702-8368-3831f717fd8e-workload-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-prccm\" (UID: \"903e1c2f-42e8-4702-8368-3831f717fd8e\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"
Oct 30 20:48:47.005458 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.005242    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd81aa48-0f88-44fb-8baa-db1deec4a2df-workload-certs\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s\" (UID: \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"
Oct 30 20:48:47.005458 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.005280    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa5037cf-aea2-403e-9865-8df55fe7e04f-workload-certs\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl\" (UID: \"aa5037cf-aea2-403e-9865-8df55fe7e04f\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"
Oct 30 20:48:47.005458 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.005315    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scxx5\" (UniqueName: \"kubernetes.io/projected/eae68bfa-6442-4208-b363-71f7953d875e-kube-api-access-scxx5\") pod \"payload-processing-e2e-derive-bd92310f-7f56b44dcd-txxfn\" (UID: \"eae68bfa-6442-4208-b363-71f7953d875e\") " pod="openshift-ingress/payload-processing-e2e-derive-bd92310f-7f56b44dcd-txxfn"
Oct 30 20:48:47.005458 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.005355    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/903e1c2f-42e8-4702-8368-3831f717fd8e-istio-envoy\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-prccm\" (UID: \"903e1c2f-42e8-4702-8368-3831f717fd8e\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"
Oct 30 20:48:47.005458 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.005389    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/903e1c2f-42e8-4702-8368-3831f717fd8e-istio-token\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-prccm\" (UID: \"903e1c2f-42e8-4702-8368-3831f717fd8e\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"
Oct 30 20:48:47.005458 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.005417    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd81aa48-0f88-44fb-8baa-db1deec4a2df-istio-podinfo\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s\" (UID: \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"
Oct 30 20:48:47.005845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.005464    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/903e1c2f-42e8-4702-8368-3831f717fd8e-istio-data\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-prccm\" (UID: \"903e1c2f-42e8-4702-8368-3831f717fd8e\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"
Oct 30 20:48:47.005845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.005729    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa5037cf-aea2-403e-9865-8df55fe7e04f-workload-certs\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl\" (UID: \"aa5037cf-aea2-403e-9865-8df55fe7e04f\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"
Oct 30 20:48:47.005845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.005467    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5037cf-aea2-403e-9865-8df55fe7e04f-workload-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl\" (UID: \"aa5037cf-aea2-403e-9865-8df55fe7e04f\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"
Oct 30 20:48:47.005949 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.005841    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/903e1c2f-42e8-4702-8368-3831f717fd8e-istiod-ca-cert\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-prccm\" (UID: \"903e1c2f-42e8-4702-8368-3831f717fd8e\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"
Oct 30 20:48:47.005949 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.005467    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6jjt\" (UniqueName: \"kubernetes.io/projected/903e1c2f-42e8-4702-8368-3831f717fd8e-kube-api-access-t6jjt\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-prccm\" (UID: \"903e1c2f-42e8-4702-8368-3831f717fd8e\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"
Oct 30 20:48:47.005949 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.005880    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/903e1c2f-42e8-4702-8368-3831f717fd8e-workload-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-prccm\" (UID: \"903e1c2f-42e8-4702-8368-3831f717fd8e\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"
Oct 30 20:48:47.005949 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.005906    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/903e1c2f-42e8-4702-8368-3831f717fd8e-credential-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-prccm\" (UID: \"903e1c2f-42e8-4702-8368-3831f717fd8e\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"
Oct 30 20:48:47.005949 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.005929    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd81aa48-0f88-44fb-8baa-db1deec4a2df-istiod-ca-cert\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s\" (UID: \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"
Oct 30 20:48:47.006237 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.005983    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa5037cf-aea2-403e-9865-8df55fe7e04f-istiod-ca-cert\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl\" (UID: \"aa5037cf-aea2-403e-9865-8df55fe7e04f\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"
Oct 30 20:48:47.006237 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.006026    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd81aa48-0f88-44fb-8baa-db1deec4a2df-istio-envoy\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s\" (UID: \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"
Oct 30 20:48:47.006237 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.006132    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd81aa48-0f88-44fb-8baa-db1deec4a2df-istio-token\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s\" (UID: \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"
Oct 30 20:48:47.006237 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.006172    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa5037cf-aea2-403e-9865-8df55fe7e04f-istio-data\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl\" (UID: \"aa5037cf-aea2-403e-9865-8df55fe7e04f\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"
Oct 30 20:48:47.006237 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.006202    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd81aa48-0f88-44fb-8baa-db1deec4a2df-workload-certs\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s\" (UID: \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"
Oct 30 20:48:47.006237 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.006216    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd81aa48-0f88-44fb-8baa-db1deec4a2df-workload-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s\" (UID: \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"
Oct 30 20:48:47.006573 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.006290    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/903e1c2f-42e8-4702-8368-3831f717fd8e-credential-socket\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-prccm\" (UID: \"903e1c2f-42e8-4702-8368-3831f717fd8e\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"
Oct 30 20:48:47.006573 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.006535    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa5037cf-aea2-403e-9865-8df55fe7e04f-istio-data\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl\" (UID: \"aa5037cf-aea2-403e-9865-8df55fe7e04f\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"
Oct 30 20:48:47.006709 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.006674    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd81aa48-0f88-44fb-8baa-db1deec4a2df-workload-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s\" (UID: \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"
Oct 30 20:48:47.006770 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.006730    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa5037cf-aea2-403e-9865-8df55fe7e04f-istio-token\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl\" (UID: \"aa5037cf-aea2-403e-9865-8df55fe7e04f\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"
Oct 30 20:48:47.006855 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.006797    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa5037cf-aea2-403e-9865-8df55fe7e04f-istio-podinfo\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl\" (UID: \"aa5037cf-aea2-403e-9865-8df55fe7e04f\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"
Oct 30 20:48:47.006921 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.006882    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd81aa48-0f88-44fb-8baa-db1deec4a2df-credential-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s\" (UID: \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"
Oct 30 20:48:47.006921 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.006907    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd81aa48-0f88-44fb-8baa-db1deec4a2df-istio-data\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s\" (UID: \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"
Oct 30 20:48:47.007042 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.006954    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eae68bfa-6442-4208-b363-71f7953d875e-plugins-config-volume\") pod \"payload-processing-e2e-derive-bd92310f-7f56b44dcd-txxfn\" (UID: \"eae68bfa-6442-4208-b363-71f7953d875e\") " pod="openshift-ingress/payload-processing-e2e-derive-bd92310f-7f56b44dcd-txxfn"
Oct 30 20:48:47.007042 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.006964    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa5037cf-aea2-403e-9865-8df55fe7e04f-istiod-ca-cert\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl\" (UID: \"aa5037cf-aea2-403e-9865-8df55fe7e04f\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"
Oct 30 20:48:47.007042 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.007025    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjxq5\" (UniqueName: \"kubernetes.io/projected/fd81aa48-0f88-44fb-8baa-db1deec4a2df-kube-api-access-hjxq5\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s\" (UID: \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"
Oct 30 20:48:47.007211 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.007193    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5037cf-aea2-403e-9865-8df55fe7e04f-credential-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl\" (UID: \"aa5037cf-aea2-403e-9865-8df55fe7e04f\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"
Oct 30 20:48:47.007277 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.007231    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa5037cf-aea2-403e-9865-8df55fe7e04f-istio-envoy\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl\" (UID: \"aa5037cf-aea2-403e-9865-8df55fe7e04f\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"
Oct 30 20:48:47.007277 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.007262    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/903e1c2f-42e8-4702-8368-3831f717fd8e-workload-certs\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-prccm\" (UID: \"903e1c2f-42e8-4702-8368-3831f717fd8e\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"
Oct 30 20:48:47.007390 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.007288    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/903e1c2f-42e8-4702-8368-3831f717fd8e-istio-podinfo\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-prccm\" (UID: \"903e1c2f-42e8-4702-8368-3831f717fd8e\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"
Oct 30 20:48:47.007390 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.007364    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lczf9\" (UniqueName: \"kubernetes.io/projected/aa5037cf-aea2-403e-9865-8df55fe7e04f-kube-api-access-lczf9\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl\" (UID: \"aa5037cf-aea2-403e-9865-8df55fe7e04f\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"
Oct 30 20:48:47.007586 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.007508    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5037cf-aea2-403e-9865-8df55fe7e04f-credential-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl\" (UID: \"aa5037cf-aea2-403e-9865-8df55fe7e04f\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"
Oct 30 20:48:47.007922 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.007698    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd81aa48-0f88-44fb-8baa-db1deec4a2df-istio-data\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s\" (UID: \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"
Oct 30 20:48:47.007922 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.007786    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/903e1c2f-42e8-4702-8368-3831f717fd8e-workload-certs\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-prccm\" (UID: \"903e1c2f-42e8-4702-8368-3831f717fd8e\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"
Oct 30 20:48:47.007922 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.007912    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eae68bfa-6442-4208-b363-71f7953d875e-plugins-config-volume\") pod \"payload-processing-e2e-derive-bd92310f-7f56b44dcd-txxfn\" (UID: \"eae68bfa-6442-4208-b363-71f7953d875e\") " pod="openshift-ingress/payload-processing-e2e-derive-bd92310f-7f56b44dcd-txxfn"
Oct 30 20:48:47.008136 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.007997    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd81aa48-0f88-44fb-8baa-db1deec4a2df-credential-socket\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s\" (UID: \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"
Oct 30 20:48:47.008701 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.008675    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd81aa48-0f88-44fb-8baa-db1deec4a2df-istio-envoy\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s\" (UID: \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"
Oct 30 20:48:47.008967 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.008948    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd81aa48-0f88-44fb-8baa-db1deec4a2df-istio-podinfo\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s\" (UID: \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"
Oct 30 20:48:47.009069 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.009045    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/903e1c2f-42e8-4702-8368-3831f717fd8e-istio-envoy\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-prccm\" (UID: \"903e1c2f-42e8-4702-8368-3831f717fd8e\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"
Oct 30 20:48:47.010027 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.010007    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa5037cf-aea2-403e-9865-8df55fe7e04f-istio-envoy\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl\" (UID: \"aa5037cf-aea2-403e-9865-8df55fe7e04f\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"
Oct 30 20:48:47.010323 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.010298    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/903e1c2f-42e8-4702-8368-3831f717fd8e-istio-podinfo\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-prccm\" (UID: \"903e1c2f-42e8-4702-8368-3831f717fd8e\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"
Oct 30 20:48:47.010530 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.010509    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa5037cf-aea2-403e-9865-8df55fe7e04f-istio-podinfo\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl\" (UID: \"aa5037cf-aea2-403e-9865-8df55fe7e04f\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"
Oct 30 20:48:47.066288 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.066251    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd81aa48-0f88-44fb-8baa-db1deec4a2df-istio-token\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s\" (UID: \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"
Oct 30 20:48:47.067220 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.067159    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6jjt\" (UniqueName: \"kubernetes.io/projected/903e1c2f-42e8-4702-8368-3831f717fd8e-kube-api-access-t6jjt\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-prccm\" (UID: \"903e1c2f-42e8-4702-8368-3831f717fd8e\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"
Oct 30 20:48:47.067352 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.067311    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scxx5\" (UniqueName: \"kubernetes.io/projected/eae68bfa-6442-4208-b363-71f7953d875e-kube-api-access-scxx5\") pod \"payload-processing-e2e-derive-bd92310f-7f56b44dcd-txxfn\" (UID: \"eae68bfa-6442-4208-b363-71f7953d875e\") " pod="openshift-ingress/payload-processing-e2e-derive-bd92310f-7f56b44dcd-txxfn"
Oct 30 20:48:47.067686 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.067644    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lczf9\" (UniqueName: \"kubernetes.io/projected/aa5037cf-aea2-403e-9865-8df55fe7e04f-kube-api-access-lczf9\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl\" (UID: \"aa5037cf-aea2-403e-9865-8df55fe7e04f\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"
Oct 30 20:48:47.067789 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.067720    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa5037cf-aea2-403e-9865-8df55fe7e04f-istio-token\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl\" (UID: \"aa5037cf-aea2-403e-9865-8df55fe7e04f\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"
Oct 30 20:48:47.067915 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.067895    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/903e1c2f-42e8-4702-8368-3831f717fd8e-istio-token\") pod \"e2e-worker-w4-395022-openshift-default-559878f94c-prccm\" (UID: \"903e1c2f-42e8-4702-8368-3831f717fd8e\") " pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"
Oct 30 20:48:47.068347 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.068328    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjxq5\" (UniqueName: \"kubernetes.io/projected/fd81aa48-0f88-44fb-8baa-db1deec4a2df-kube-api-access-hjxq5\") pod \"e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s\" (UID: \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\") " pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"
Oct 30 20:48:47.075365 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.075346    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"
Oct 30 20:48:47.104516 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.104489    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"
Oct 30 20:48:47.117565 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.117540    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"
Oct 30 20:48:47.155160 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.155129    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bd92310f-d9b4b64d7-pbvp7"]
Oct 30 20:48:47.165997 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.162422    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"]
Oct 30 20:48:47.165997 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:48:47.164044    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod576450be_aec9_44a7_a91b_b2bd82342e57.slice/crio-2def2ddde70cfcb0dd341f0a5e74178d33f4fb4368c1dc8f184dfed821b2cafb WatchSource:0}: Error finding container 2def2ddde70cfcb0dd341f0a5e74178d33f4fb4368c1dc8f184dfed821b2cafb: Status 404 returned error can't find the container with id 2def2ddde70cfcb0dd341f0a5e74178d33f4fb4368c1dc8f184dfed821b2cafb
Oct 30 20:48:47.165997 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.164180    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:48:47.165997 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.164239    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:48:47.165997 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.164266    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:48:47.351106 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.350989    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-bd92310f-7f56b44dcd-txxfn"
Oct 30 20:48:47.475543 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.475500    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"]
Oct 30 20:48:47.476144 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:48:47.476023    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa5037cf_aea2_403e_9865_8df55fe7e04f.slice/crio-77c65115e69335137d7f32fdadccc40c770c29870f1ee1fa844d54b800e45eee WatchSource:0}: Error finding container 77c65115e69335137d7f32fdadccc40c770c29870f1ee1fa844d54b800e45eee: Status 404 returned error can't find the container with id 77c65115e69335137d7f32fdadccc40c770c29870f1ee1fa844d54b800e45eee
Oct 30 20:48:47.482867 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.480298    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:48:47.482867 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.480404    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:48:47.482867 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.480442    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:48:47.527526 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.526388    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"]
Oct 30 20:48:47.538015 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:48:47.537087    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd81aa48_0f88_44fb_8baa_db1deec4a2df.slice/crio-5e34b39f6d58daccbf7cc7cede198d7d0df97d895d6b745fc48e3daeff716f0b WatchSource:0}: Error finding container 5e34b39f6d58daccbf7cc7cede198d7d0df97d895d6b745fc48e3daeff716f0b: Status 404 returned error can't find the container with id 5e34b39f6d58daccbf7cc7cede198d7d0df97d895d6b745fc48e3daeff716f0b
Oct 30 20:48:47.542210 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.542165    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:48:47.542567 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.542537    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:48:47.542759 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.542739    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:48:47.597118 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.595422    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"]
Oct 30 20:48:47.597118 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:48:47.595882    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod903e1c2f_42e8_4702_8368_3831f717fd8e.slice/crio-a8f8efcf70c2006488051e77bab41cd6e5104281b57c69faecd4abb6e95d5518 WatchSource:0}: Error finding container a8f8efcf70c2006488051e77bab41cd6e5104281b57c69faecd4abb6e95d5518: Status 404 returned error can't find the container with id a8f8efcf70c2006488051e77bab41cd6e5104281b57c69faecd4abb6e95d5518
Oct 30 20:48:47.600591 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.599669    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:48:47.600591 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.599743    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:48:47.600591 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.599792    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:48:47.794003 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.793971    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bd92310f-7f56b44dcd-txxfn"]
Oct 30 20:48:47.797969 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:48:47.797932    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae68bfa_6442_4208_b363_71f7953d875e.slice/crio-ea8737b6b3d1cb30302d1fb3e999b31d9075e94851575b15ff56e5ce84a62822 WatchSource:0}: Error finding container ea8737b6b3d1cb30302d1fb3e999b31d9075e94851575b15ff56e5ce84a62822: Status 404 returned error can't find the container with id ea8737b6b3d1cb30302d1fb3e999b31d9075e94851575b15ff56e5ce84a62822
Oct 30 20:48:47.990931 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.990124    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bd92310f-d9b4b64d7-pbvp7" event={"ID":"576450be-aec9-44a7-a91b-b2bd82342e57","Type":"ContainerStarted","Data":"40c636166e5358ce0f65d91b8c279afc9b5a229fa137e0732e78889cf81781d4"}
Oct 30 20:48:47.990931 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.990236    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bd92310f-d9b4b64d7-pbvp7" event={"ID":"576450be-aec9-44a7-a91b-b2bd82342e57","Type":"ContainerStarted","Data":"2def2ddde70cfcb0dd341f0a5e74178d33f4fb4368c1dc8f184dfed821b2cafb"}
Oct 30 20:48:47.992712 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.991620    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-bd92310f-d9b4b64d7-pbvp7"
Oct 30 20:48:47.998846 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.994482    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92" event={"ID":"31b52815-7184-4c7b-a294-a2d9a7d0c099","Type":"ContainerStarted","Data":"67664ec9a132ab24151dca469320d3494a4f9480c3c446b816cd116e47234e06"}
Oct 30 20:48:47.998846 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.994530    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92" event={"ID":"31b52815-7184-4c7b-a294-a2d9a7d0c099","Type":"ContainerStarted","Data":"8aee0e0b76a26a8fce3b1015c7a77df189bf0e4d95fbd6088c4280eb931051c4"}
Oct 30 20:48:47.998846 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.997994    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm" event={"ID":"903e1c2f-42e8-4702-8368-3831f717fd8e","Type":"ContainerStarted","Data":"5ceeab6e98675cc7f87f3a1923620bc5c46f692c3b6077a80eeccfc61a22b189"}
Oct 30 20:48:47.998846 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:47.998054    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm" event={"ID":"903e1c2f-42e8-4702-8368-3831f717fd8e","Type":"ContainerStarted","Data":"a8f8efcf70c2006488051e77bab41cd6e5104281b57c69faecd4abb6e95d5518"}
Oct 30 20:48:48.015873 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:48.015406    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-bd92310f-7f56b44dcd-txxfn" event={"ID":"eae68bfa-6442-4208-b363-71f7953d875e","Type":"ContainerStarted","Data":"ea8737b6b3d1cb30302d1fb3e999b31d9075e94851575b15ff56e5ce84a62822"}
Oct 30 20:48:48.037063 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:48.037015    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl" event={"ID":"aa5037cf-aea2-403e-9865-8df55fe7e04f","Type":"ContainerStarted","Data":"4dab2753431dab7f9c6ebd9aae38bb413c72553660c8459a4ab0db453edc091a"}
Oct 30 20:48:48.037322 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:48.037302    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl" event={"ID":"aa5037cf-aea2-403e-9865-8df55fe7e04f","Type":"ContainerStarted","Data":"77c65115e69335137d7f32fdadccc40c770c29870f1ee1fa844d54b800e45eee"}
Oct 30 20:48:48.040305 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:48.040279    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s" event={"ID":"fd81aa48-0f88-44fb-8baa-db1deec4a2df","Type":"ContainerStarted","Data":"753e257909c6531af48f1cafec5c39fb421f9e97fede074c88bf601b9071eb0f"}
Oct 30 20:48:48.040482 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:48.040465    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s" event={"ID":"fd81aa48-0f88-44fb-8baa-db1deec4a2df","Type":"ContainerStarted","Data":"5e34b39f6d58daccbf7cc7cede198d7d0df97d895d6b745fc48e3daeff716f0b"}
Oct 30 20:48:48.076845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:48.076385    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"
Oct 30 20:48:48.089557 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:48.088281    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 20:48:48.089557 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:48.088339    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl" podUID="aa5037cf-aea2-403e-9865-8df55fe7e04f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 20:48:48.109604 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:48.108175    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"
Oct 30 20:48:48.109604 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:48.109216    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-bd92310f-d9b4b64d7-pbvp7" podStartSLOduration=3.109197192 podStartE2EDuration="3.109197192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:48.107315293 +0000 UTC m=+1340.024336851" watchObservedRunningTime="2026-10-30 20:48:48.109197192 +0000 UTC m=+1340.026218729"
Oct 30 20:48:48.109604 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:48.109471    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:48:48.109604 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:48.109521    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s" podUID="fd81aa48-0f88-44fb-8baa-db1deec4a2df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 20:48:48.119079 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:48.118904    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"
Oct 30 20:48:48.119515 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:48.119406    5071 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-prccm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 20:48:48.119515 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:48.119458    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm" podUID="903e1c2f-42e8-4702-8368-3831f717fd8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 20:48:48.231465 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:48.230750    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s" podStartSLOduration=2.230729028 podStartE2EDuration="2.230729028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:48.228375052 +0000 UTC m=+1340.145396614" watchObservedRunningTime="2026-10-30 20:48:48.230729028 +0000 UTC m=+1340.147750578"
Oct 30 20:48:48.352121 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:48.352094    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-dc0e9f64-66454765d-kn2vl"
Oct 30 20:48:48.455907 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:48.443762    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm" podStartSLOduration=2.443742362 podStartE2EDuration="2.443742362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:48.349386669 +0000 UTC m=+1340.266408205" watchObservedRunningTime="2026-10-30 20:48:48.443742362 +0000 UTC m=+1340.360763899"
Oct 30 20:48:48.488148 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:48.488090    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92" podStartSLOduration=3.488070391 podStartE2EDuration="3.488070391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:48.474570609 +0000 UTC m=+1340.391592144" watchObservedRunningTime="2026-10-30 20:48:48.488070391 +0000 UTC m=+1340.405091929"
Oct 30 20:48:48.618248 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:48.594890    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"
Oct 30 20:48:48.618248 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:48.595046    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 30 20:48:48.618248 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:48.595092    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92" podUID="31b52815-7184-4c7b-a294-a2d9a7d0c099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 30 20:48:48.623579 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:48.623475    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl" podStartSLOduration=2.623423603 podStartE2EDuration="2.623423603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:48.603651726 +0000 UTC m=+1340.520673265" watchObservedRunningTime="2026-10-30 20:48:48.623423603 +0000 UTC m=+1340.540445228"
Oct 30 20:48:49.223334 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:49.173960    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:48:49.223334 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:49.174030    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s" podUID="fd81aa48-0f88-44fb-8baa-db1deec4a2df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 20:48:49.608085 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:49.608038    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 30 20:48:49.608431 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:49.608395    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92" podUID="31b52815-7184-4c7b-a294-a2d9a7d0c099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 30 20:48:49.625847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:49.622408    5071 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-prccm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 20:48:49.629961 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:49.629921    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm" podUID="903e1c2f-42e8-4702-8368-3831f717fd8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 20:48:49.651959 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:49.626815    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:49.655987 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:49.625932    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 20:48:49.656214 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:49.656190    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl" podUID="aa5037cf-aea2-403e-9865-8df55fe7e04f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 20:48:50.090870 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:50.078152    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 20:48:50.090870 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:50.078214    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl" podUID="aa5037cf-aea2-403e-9865-8df55fe7e04f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 20:48:50.119371 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:50.119334    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:48:50.156976 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:50.156925    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s" podUID="fd81aa48-0f88-44fb-8baa-db1deec4a2df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 20:48:50.157286 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:50.134433    5071 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-prccm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 20:48:50.157429 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:50.157404    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm" podUID="903e1c2f-42e8-4702-8368-3831f717fd8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 20:48:50.177055 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:50.177003    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:50.203338 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:50.198571    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:50.204146 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:50.198625    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 30 20:48:50.215075 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:50.208669    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:50.450271 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:50.450155    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-bd92310f-7f56b44dcd-txxfn" podStartSLOduration=4.450135362 podStartE2EDuration="4.450135362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:50.33311978 +0000 UTC m=+1342.250141319" watchObservedRunningTime="2026-10-30 20:48:50.450135362 +0000 UTC m=+1342.367156900"
Oct 30 20:48:50.459936 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:50.450588    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:50.520850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:50.520202    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-bd92310f-7f56b44dcd-txxfn" event={"ID":"eae68bfa-6442-4208-b363-71f7953d875e","Type":"ContainerStarted","Data":"6d43c1fa71dbf17aa4426766b4bd51c0616d239806d37de10d0fa18ae00efa27"}
Oct 30 20:48:50.520850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:50.520356    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-bd92310f-7f56b44dcd-txxfn"
Oct 30 20:48:50.594922 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:50.594879    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 30 20:48:50.595204 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:50.595173    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92" podUID="31b52815-7184-4c7b-a294-a2d9a7d0c099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 30 20:48:50.607753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:50.607708    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:50.685435 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:50.685401    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:51.133866 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:51.126252    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:48:51.133866 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:51.126315    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s" podUID="fd81aa48-0f88-44fb-8baa-db1deec4a2df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 20:48:51.133866 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:51.126987    5071 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-prccm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 20:48:51.133866 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:51.127026    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm" podUID="903e1c2f-42e8-4702-8368-3831f717fd8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 20:48:51.133866 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:51.128421    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 20:48:51.133866 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:51.128457    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl" podUID="aa5037cf-aea2-403e-9865-8df55fe7e04f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 20:48:51.208850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:51.205006    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:51.208850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:51.205438    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 20:48:51.208850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:51.205695    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:51.229854 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:51.210954    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:48:51.229854 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:51.211017    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:48:51.229854 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:51.211060    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:48:51.229854 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:51.216941    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:51.229854 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:51.217362    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:48:51.248856 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:51.243088    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:51.628004 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:51.627966    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 30 20:48:51.628641 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:51.628600    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92" podUID="31b52815-7184-4c7b-a294-a2d9a7d0c099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 30 20:48:51.629729 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:51.629700    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:51.630246 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:51.630227    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:48:52.168847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:52.168388    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 20:48:52.168847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:52.168445    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl" podUID="aa5037cf-aea2-403e-9865-8df55fe7e04f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 20:48:52.211857 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:52.210705    5071 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-prccm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 20:48:52.211857 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:52.210789    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm" podUID="903e1c2f-42e8-4702-8368-3831f717fd8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 20:48:52.211857 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:52.211804    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:48:52.212185 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:52.211865    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s" podUID="fd81aa48-0f88-44fb-8baa-db1deec4a2df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 20:48:52.258297 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:52.224963    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:52.258297 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:52.254231    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:52.281042 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:52.280994    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 30 20:48:52.281580 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:52.281547    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 20:48:52.564909 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:52.533203    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:48:52.564909 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:52.533267    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:48:52.630847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:52.630714    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:52.717849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:52.698593    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 30 20:48:52.717849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:52.698669    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92" podUID="31b52815-7184-4c7b-a294-a2d9a7d0c099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 30 20:48:53.144443 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:53.144405    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 20:48:53.166966 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:53.166913    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl" podUID="aa5037cf-aea2-403e-9865-8df55fe7e04f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 20:48:53.275933 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:53.255276    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:53.432914 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:53.295541    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:53.695303 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:53.316541    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:48:53.695930 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:53.695885    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s" podUID="fd81aa48-0f88-44fb-8baa-db1deec4a2df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 20:48:53.696057 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:53.671975    5071 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:53.699857 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:53.634251    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:53.700327 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:53.430952    5071 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-prccm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 20:48:53.700506 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:53.700480    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm" podUID="903e1c2f-42e8-4702-8368-3831f717fd8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 20:48:53.716944 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:53.634189    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 30 20:48:53.717153 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:53.717126    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92" podUID="31b52815-7184-4c7b-a294-a2d9a7d0c099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 30 20:48:54.096026 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:54.095984    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 20:48:54.096334 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:54.096303    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl" podUID="aa5037cf-aea2-403e-9865-8df55fe7e04f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 20:48:54.127694 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:54.127649    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:48:54.133965 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:54.133927    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s" podUID="fd81aa48-0f88-44fb-8baa-db1deec4a2df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 20:48:54.134194 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:54.133079    5071 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-prccm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 20:48:54.134325 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:54.134309    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm" podUID="903e1c2f-42e8-4702-8368-3831f717fd8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 20:48:54.158764 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:54.158730    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:54.305387 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:54.305242    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 30 20:48:54.409050 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:54.408947    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:54.436861 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:54.435933    5071 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:54.684906 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:54.629212    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 30 20:48:54.684906 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:54.629270    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92" podUID="31b52815-7184-4c7b-a294-a2d9a7d0c099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 30 20:48:54.699590 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:54.699175    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:54.737336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:54.700937    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:55.087197 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:55.082615    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 20:48:55.087197 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:55.082684    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl" podUID="aa5037cf-aea2-403e-9865-8df55fe7e04f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 20:48:55.183439 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:55.173584    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:55.200454 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:55.188604    5071 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-prccm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 20:48:55.200454 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:55.188692    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm" podUID="903e1c2f-42e8-4702-8368-3831f717fd8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 20:48:55.239987 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:55.239948    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:48:55.240260 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:55.240230    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s" podUID="fd81aa48-0f88-44fb-8baa-db1deec4a2df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 20:48:55.512903 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:55.383675    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 30 20:48:55.549843 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:55.521560    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:48:55.665881 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:55.544492    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 20:48:55.666130 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:55.544538    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:55.666563 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:55.594638    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 30 20:48:55.666741 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:55.666714    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92" podUID="31b52815-7184-4c7b-a294-a2d9a7d0c099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 30 20:48:55.726980 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:55.714310    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:55.726980 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:55.715009    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 20:48:56.105420 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:56.105386    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:48:56.105728 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:56.105685    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s" podUID="fd81aa48-0f88-44fb-8baa-db1deec4a2df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 20:48:56.112669 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:56.106278    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 20:48:56.112951 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:56.112904    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl" podUID="aa5037cf-aea2-403e-9865-8df55fe7e04f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 20:48:56.135048 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:56.135014    5071 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-prccm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 20:48:56.157269 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:56.157189    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm" podUID="903e1c2f-42e8-4702-8368-3831f717fd8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 20:48:56.524901 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:56.524230    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 30 20:48:56.591847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:56.590423    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 30 20:48:56.591847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:56.590484    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92" podUID="31b52815-7184-4c7b-a294-a2d9a7d0c099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 30 20:48:56.672853 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:56.669143    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:56.786750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:56.721975    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 20:48:56.803553 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:56.722023    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:56.815949 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:56.748639    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:56.815949 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:56.804743    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:48:57.014966 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:57.014919    5071 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:57.117797 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:57.117573    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:48:57.118143 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:57.118088    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s" podUID="fd81aa48-0f88-44fb-8baa-db1deec4a2df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 20:48:57.118621 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:57.118585    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 20:48:57.118739 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:57.118650    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl" podUID="aa5037cf-aea2-403e-9865-8df55fe7e04f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 20:48:57.204706 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:57.204663    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-78ca07a3-db58645f-ckkt9"
Oct 30 20:48:57.206797 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:57.206778    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-78ca07a3-6f8759c9f8-xz5m9"
Oct 30 20:48:57.211380 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:57.211357    5071 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-prccm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 20:48:57.211578 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:57.211542    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm" podUID="903e1c2f-42e8-4702-8368-3831f717fd8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 20:48:57.324867 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:57.324837    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:48:57.397790 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:57.397684    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:48:57.406437 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:57.406395    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:48:57.485514 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:57.485483    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:48:57.487197 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:57.487142    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:48:57.600783 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:57.600625    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 30 20:48:57.600783 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:57.600683    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92" podUID="31b52815-7184-4c7b-a294-a2d9a7d0c099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 30 20:48:58.097217 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:58.096355    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 20:48:58.097217 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:58.096416    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl" podUID="aa5037cf-aea2-403e-9865-8df55fe7e04f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 20:48:58.119242 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:58.118476    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:48:58.119242 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:58.118532    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s" podUID="fd81aa48-0f88-44fb-8baa-db1deec4a2df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 20:48:58.119242 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:58.118930    5071 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-prccm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 20:48:58.119242 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:58.118967    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm" podUID="903e1c2f-42e8-4702-8368-3831f717fd8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 20:48:58.182977 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:58.182864    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:58.466200 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:58.463894    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:48:58.792040 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:58.790331    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"
Oct 30 20:48:58.792913 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:58.792886    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"
Oct 30 20:48:58.795582 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:58.795433    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"
Oct 30 20:48:59.185957 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:59.177661    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"
Oct 30 20:48:59.185957 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:59.179320    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"
Oct 30 20:48:59.203035 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:59.199643    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"
Oct 30 20:48:59.265905 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:59.264297    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"
Oct 30 20:48:59.266374 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:59.266213    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"
Oct 30 20:48:59.274808 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:59.274751    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"
Oct 30 20:48:59.515883 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:59.515698    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"
Oct 30 20:48:59.517056 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:59.516189    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"
Oct 30 20:48:59.517809 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:48:59.517791    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"
Oct 30 20:49:00.577055 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:00.576980    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:01.675208 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:01.675173    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-bd92310f-7f56b44dcd-txxfn"
Oct 30 20:49:01.683259 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:01.683232    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-bd92310f-d9b4b64d7-pbvp7"
Oct 30 20:49:02.587505 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:02.587470    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:49:04.635984 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:04.635932    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 20:49:04.670463 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:04.670417    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:49:05.332425 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:05.332378    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:49:05.647904 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:05.637908    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:49:05.647904 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:05.638422    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:49:05.670947 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:05.670904    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:49:05.673203 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:05.671711    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:49:06.437637 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:06.437593    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 20:49:06.526849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:06.514481    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:49:06.551862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:06.544918    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:49:06.670518 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:06.670470    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 20:49:06.673954 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:06.673915    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 20:49:06.836865 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:06.834465    5071 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:49:06.836865 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:06.834534    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t" podUID="00b1175c-ea5d-49a8-b721-93fd709970c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:49:07.145235 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:07.145124    5071 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:49:07.273848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:07.273741    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:07.331849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:07.305039    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:49:07.440956 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:07.390039    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:49:07.479856 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:07.471162    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g" podUID="05a72fc0-2554-4f73-935e-ade8fb962a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:49:07.480125 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:07.439975    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:49:07.560848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:07.550945    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 30 20:49:07.560848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:07.551444    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:49:07.718912 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:07.710932    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 20:49:07.778733 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:07.717599    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:49:07.799313 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:07.799275    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87" podUID="723f1905-6d5a-45a7-a2bb-d5753a0fda9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:49:07.799582 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:07.729571    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 20:49:08.291851 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:08.273741    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:08.464851 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:08.459686    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:49:08.514849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:08.502966    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:49:08.558464 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:08.557924    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:49:08.915392 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:08.915298    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 20:49:08.966847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:08.929103    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:49:08.968249 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:08.945722    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:49:09.116432 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:09.116381    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:49:09.149305 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:09.117435    5071 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:49:09.246855 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:09.246336    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:49:09.254400 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:09.254372    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:49:09.331939 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:09.331905    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:49:09.407720 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:09.407691    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:49:13.795253 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:13.795211    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:16.729629 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:49:16.729593    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:49:18.655050 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:18.655002    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:18.835848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:18.835699    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:19.014899 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:19.014563    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:23.582149 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:23.582111    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:28.622439 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:28.622407    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:28.807074 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:28.807041    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:29.002573 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:29.002459    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:29.679519 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:29.679481    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:32.396656 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:32.396617    5071 scope.go:122] "RemoveContainer" containerID="ec4695415cdb07486a9b8b03e436548417b31ab4a9b6ee3cfc9f378d6a8e9f29"
Oct 30 20:49:32.408461 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:32.408435    5071 scope.go:122] "RemoveContainer" containerID="1062ca2b6c343e9cc24e3c121c593c1d78ccbfe7fa3882af4157ba8a1bbd6797"
Oct 30 20:49:32.420135 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:32.420107    5071 scope.go:122] "RemoveContainer" containerID="a27d74b5e77ecbf03cd816050d9ba51526ea55fdb55428abc2b8b4a4c86bc0b3"
Oct 30 20:49:32.509374 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:32.509339    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:33.834000 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:33.833958    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:34.066025 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:34.065987    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:34.240947 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:34.240857    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:34.910284 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:34.910240    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:39.085865 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:39.085736    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:39.323538 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:39.323487    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:39.504126 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:39.504036    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:44.416024 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:44.415948    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:46.730161 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:49:46.730119    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:49:49.626844 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:49.622707    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:54.685812 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:54.685776    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:49:59.839847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:49:59.839342    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:04.808126 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:04.808086    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:13.010731 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:13.010695    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:14.977758 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:14.977722    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:16.523114 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:16.523079    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:50:16.709477 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:16.709436    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:16.908209 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:16.908178    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:16.928165 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:50:16.928106    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:50:18.353534 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:18.353439    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:18.594963 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:18.594927    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:20.886505 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:20.886371    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:25.841853 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:25.839630    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:26.004892 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:26.004855    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-939d410e-twdgv"]
Oct 30 20:50:26.010209 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:26.010184    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-939d410e-twdgv"
Oct 30 20:50:26.072710 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:26.072676    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-939d410e-twdgv"]
Oct 30 20:50:26.083141 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:26.083094    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-939d410e-twdgv\" (UID: \"c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-939d410e-twdgv"
Oct 30 20:50:26.083329 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:26.083223    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqj98\" (UniqueName: \"kubernetes.io/projected/c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd-kube-api-access-vqj98\") pod \"maas-api-revoke-sub-939d410e-twdgv\" (UID: \"c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-939d410e-twdgv"
Oct 30 20:50:26.083329 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:26.083297    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-939d410e-twdgv\" (UID: \"c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-939d410e-twdgv"
Oct 30 20:50:26.114161 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:26.114056    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:50:26.114436 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:26.114164    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 20:50:26.185138 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:26.185089    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-939d410e-twdgv\" (UID: \"c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-939d410e-twdgv"
Oct 30 20:50:26.185340 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:26.185154    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqj98\" (UniqueName: \"kubernetes.io/projected/c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd-kube-api-access-vqj98\") pod \"maas-api-revoke-sub-939d410e-twdgv\" (UID: \"c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-939d410e-twdgv"
Oct 30 20:50:26.185340 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:26.185207    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-939d410e-twdgv\" (UID: \"c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-939d410e-twdgv"
Oct 30 20:50:26.185939 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:26.185905    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-939d410e-twdgv\" (UID: \"c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-939d410e-twdgv"
Oct 30 20:50:26.188274 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:26.188223    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-939d410e-twdgv\" (UID: \"c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-939d410e-twdgv"
Oct 30 20:50:26.245077 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:26.245041    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqj98\" (UniqueName: \"kubernetes.io/projected/c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd-kube-api-access-vqj98\") pod \"maas-api-revoke-sub-939d410e-twdgv\" (UID: \"c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-939d410e-twdgv"
Oct 30 20:50:26.329181 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:26.329145    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-939d410e-twdgv"
Oct 30 20:50:26.743495 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:50:26.743447    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc71df5c9_8eae_47f4_bbbb_2c1ddb1697dd.slice/crio-913d81d8aea011d630357fa3b28eea4b0542af58e6e76268fcc9ed7a817e44fe WatchSource:0}: Error finding container 913d81d8aea011d630357fa3b28eea4b0542af58e6e76268fcc9ed7a817e44fe: Status 404 returned error can't find the container with id 913d81d8aea011d630357fa3b28eea4b0542af58e6e76268fcc9ed7a817e44fe
Oct 30 20:50:26.744488 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:26.744464    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-939d410e-twdgv"]
Oct 30 20:50:27.334022 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:27.333924    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-939d410e-twdgv" event={"ID":"c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd","Type":"ContainerStarted","Data":"aa56c88aa32cb42c8face068e993bc20f28cf8122db9adbe39a3fe0a18eae22d"}
Oct 30 20:50:27.334022 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:27.333983    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-939d410e-twdgv" event={"ID":"c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd","Type":"ContainerStarted","Data":"913d81d8aea011d630357fa3b28eea4b0542af58e6e76268fcc9ed7a817e44fe"}
Oct 30 20:50:27.452113 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:27.452056    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-939d410e-twdgv" podStartSLOduration=2.452038121 podStartE2EDuration="2.452038121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:27.450101056 +0000 UTC m=+1439.367122594" watchObservedRunningTime="2026-10-30 20:50:27.452038121 +0000 UTC m=+1439.369059659"
Oct 30 20:50:28.351707 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:28.351663    5071 generic.go:356] "Generic (PLEG): container finished" podID="c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd" containerID="aa56c88aa32cb42c8face068e993bc20f28cf8122db9adbe39a3fe0a18eae22d" exitCode=0
Oct 30 20:50:28.352245 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:28.351727    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-939d410e-twdgv" event={"ID":"c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd","Type":"ContainerDied","Data":"aa56c88aa32cb42c8face068e993bc20f28cf8122db9adbe39a3fe0a18eae22d"}
Oct 30 20:50:29.363206 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:29.363173    5071 generic.go:356] "Generic (PLEG): container finished" podID="04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d" containerID="1324533b9d510d2354a3ddc1c62a37116bdc450c04ea33254a87e8877d57ea40" exitCode=0
Oct 30 20:50:29.363206 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:29.363202    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c" event={"ID":"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d","Type":"ContainerDied","Data":"1324533b9d510d2354a3ddc1c62a37116bdc450c04ea33254a87e8877d57ea40"}
Oct 30 20:50:29.363799 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:29.363673    5071 scope.go:122] "RemoveContainer" containerID="1324533b9d510d2354a3ddc1c62a37116bdc450c04ea33254a87e8877d57ea40"
Oct 30 20:50:29.365422 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:29.365396    5071 generic.go:356] "Generic (PLEG): container finished" podID="e5944af0-5719-461a-b5bc-0c05023e086d" containerID="33f14c5bd5c7dcf3b1362429cc58fe718aecc5a65d4d44a978c8461a48ed0df5" exitCode=0
Oct 30 20:50:29.365556 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:29.365441    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:50:29.365556 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:29.365477    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7" event={"ID":"e5944af0-5719-461a-b5bc-0c05023e086d","Type":"ContainerDied","Data":"33f14c5bd5c7dcf3b1362429cc58fe718aecc5a65d4d44a978c8461a48ed0df5"}
Oct 30 20:50:29.365556 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:29.365513    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:50:29.365556 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:29.365549    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:50:29.365859 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:29.365840    5071 scope.go:122] "RemoveContainer" containerID="33f14c5bd5c7dcf3b1362429cc58fe718aecc5a65d4d44a978c8461a48ed0df5"
Oct 30 20:50:29.367461 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:29.367429    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:50:29.367563 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:29.367510    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:50:29.367563 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:29.367554    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:50:29.635914 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:29.634791    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-939d410e-twdgv"
Oct 30 20:50:29.761531 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:29.761402    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd-maas-api-xks-ca\") pod \"c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd\" (UID: \"c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd\") "
Oct 30 20:50:29.762168 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:29.762144    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd-maas-api-openshift-ca\") pod \"c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd\" (UID: \"c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd\") "
Oct 30 20:50:29.762733 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:29.762625    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd-maas-api-openshift-ca" pod "c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd" (UID: "c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:50:29.762964 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:29.762947    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd-kube-api-access-vqj98\" (UniqueName: \"kubernetes.io/projected/c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd-kube-api-access-vqj98\") pod \"c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd\" (UID: \"c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd\") "
Oct 30 20:50:29.771856 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:29.766141    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd-maas-api-xks-ca" pod "c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd" (UID: "c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:50:29.832018 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:29.831974    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd-kube-api-access-vqj98" pod "c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd" (UID: "c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd"). InnerVolumeSpecName "kube-api-access-vqj98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:50:29.892357 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:29.891863    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:50:29.892357 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:29.891901    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:50:29.892357 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:29.891920    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqj98\" (UniqueName: \"kubernetes.io/projected/c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd-kube-api-access-vqj98\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:50:30.131140 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:30.131098    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:30.208243 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:30.208208    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc0e9f64-d2s85"]
Oct 30 20:50:30.209051 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:30.209027    5071 cpu_manager.go:414] "containerMap: removing container" podUID="c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd" containerName="revoke-keys"
Oct 30 20:50:30.209051 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:30.209046    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd" containerName="revoke-keys"
Oct 30 20:50:30.209249 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:30.209160    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd" containerName="revoke-keys"
Oct 30 20:50:30.213058 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:30.213039    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc0e9f64-d2s85"
Oct 30 20:50:30.277709 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:30.277656    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc0e9f64-d2s85"]
Oct 30 20:50:30.378463 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:30.378427    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7" event={"ID":"e5944af0-5719-461a-b5bc-0c05023e086d","Type":"ContainerStarted","Data":"18cf42390ee7fb76940ea4c26bf7479fe97b6ccda9683fb11884cf0a6f1cbaf6"}
Oct 30 20:50:30.380239 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:30.380216    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-939d410e-twdgv"
Oct 30 20:50:30.380370 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:30.380250    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-939d410e-twdgv" event={"ID":"c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd","Type":"ContainerDied","Data":"913d81d8aea011d630357fa3b28eea4b0542af58e6e76268fcc9ed7a817e44fe"}
Oct 30 20:50:30.380370 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:30.380287    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="913d81d8aea011d630357fa3b28eea4b0542af58e6e76268fcc9ed7a817e44fe"
Oct 30 20:50:30.382913 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:30.382850    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c" event={"ID":"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d","Type":"ContainerStarted","Data":"ccfcd46ef63493dcb50fc6a6a0db30229b5804351dc462cd53c2019b2bb3d3c5"}
Oct 30 20:50:30.400064 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:30.400039    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da871b53-aae6-4363-95bb-78bca2b3d48f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dc0e9f64-d2s85\" (UID: \"da871b53-aae6-4363-95bb-78bca2b3d48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc0e9f64-d2s85"
Oct 30 20:50:30.400192 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:30.400087    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da871b53-aae6-4363-95bb-78bca2b3d48f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dc0e9f64-d2s85\" (UID: \"da871b53-aae6-4363-95bb-78bca2b3d48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc0e9f64-d2s85"
Oct 30 20:50:30.400355 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:30.400335    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhx7s\" (UniqueName: \"kubernetes.io/projected/da871b53-aae6-4363-95bb-78bca2b3d48f-kube-api-access-hhx7s\") pod \"maas-api-revoke-keys-e2e-migrate-dc0e9f64-d2s85\" (UID: \"da871b53-aae6-4363-95bb-78bca2b3d48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc0e9f64-d2s85"
Oct 30 20:50:30.507778 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:30.507733    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhx7s\" (UniqueName: \"kubernetes.io/projected/da871b53-aae6-4363-95bb-78bca2b3d48f-kube-api-access-hhx7s\") pod \"maas-api-revoke-keys-e2e-migrate-dc0e9f64-d2s85\" (UID: \"da871b53-aae6-4363-95bb-78bca2b3d48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc0e9f64-d2s85"
Oct 30 20:50:30.509642 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:30.508052    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da871b53-aae6-4363-95bb-78bca2b3d48f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dc0e9f64-d2s85\" (UID: \"da871b53-aae6-4363-95bb-78bca2b3d48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc0e9f64-d2s85"
Oct 30 20:50:30.509642 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:30.508117    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da871b53-aae6-4363-95bb-78bca2b3d48f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dc0e9f64-d2s85\" (UID: \"da871b53-aae6-4363-95bb-78bca2b3d48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc0e9f64-d2s85"
Oct 30 20:50:30.509642 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:30.508955    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da871b53-aae6-4363-95bb-78bca2b3d48f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dc0e9f64-d2s85\" (UID: \"da871b53-aae6-4363-95bb-78bca2b3d48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc0e9f64-d2s85"
Oct 30 20:50:30.513009 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:30.512982    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da871b53-aae6-4363-95bb-78bca2b3d48f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dc0e9f64-d2s85\" (UID: \"da871b53-aae6-4363-95bb-78bca2b3d48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc0e9f64-d2s85"
Oct 30 20:50:30.575878 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:30.575599    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhx7s\" (UniqueName: \"kubernetes.io/projected/da871b53-aae6-4363-95bb-78bca2b3d48f-kube-api-access-hhx7s\") pod \"maas-api-revoke-keys-e2e-migrate-dc0e9f64-d2s85\" (UID: \"da871b53-aae6-4363-95bb-78bca2b3d48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc0e9f64-d2s85"
Oct 30 20:50:30.836712 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:30.834147    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc0e9f64-d2s85"
Oct 30 20:50:31.105741 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:31.105572    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:50:31.107202 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:31.107133    5071 patch_prober.go:29] interesting pod/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 20:50:31.107430 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:31.107397    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c" podUID="04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 20:50:31.205598 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:31.205557    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:31.306234 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:31.306200    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:50:31.307257 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:31.307226    5071 patch_prober.go:29] interesting pod/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 20:50:31.307375 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:31.307274    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7" podUID="e5944af0-5719-461a-b5bc-0c05023e086d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 20:50:31.399924 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:31.399890    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc0e9f64-d2s85"]
Oct 30 20:50:31.529848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:31.526256    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:32.073848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:32.073746    5071 patch_prober.go:29] interesting pod/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 20:50:32.073848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:32.073806    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c" podUID="04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 20:50:32.306802 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:32.306767    5071 patch_prober.go:29] interesting pod/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 20:50:32.307004 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:32.306836    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7" podUID="e5944af0-5719-461a-b5bc-0c05023e086d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 20:50:32.424137 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:32.424048    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc0e9f64-d2s85" event={"ID":"da871b53-aae6-4363-95bb-78bca2b3d48f","Type":"ContainerStarted","Data":"ce46d77cd42b8fdeb83e37a9b9001432751063d511835f65e58f6bd905bc0cf8"}
Oct 30 20:50:32.424137 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:32.424112    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc0e9f64-d2s85" event={"ID":"da871b53-aae6-4363-95bb-78bca2b3d48f","Type":"ContainerStarted","Data":"943a0736748247231ab05c9768b8945ea73f3e8a14e2d44f969855a70b552a08"}
Oct 30 20:50:32.596290 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:32.596096    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc0e9f64-d2s85" podStartSLOduration=2.596073459 podStartE2EDuration="2.596073459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:32.589150839 +0000 UTC m=+1444.506172376" watchObservedRunningTime="2026-10-30 20:50:32.596073459 +0000 UTC m=+1444.513094997"
Oct 30 20:50:33.078899 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:33.078240    5071 patch_prober.go:29] interesting pod/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 20:50:33.078899 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:33.078300    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c" podUID="04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 20:50:33.307038 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:33.307003    5071 patch_prober.go:29] interesting pod/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 20:50:33.307216 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:33.307053    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7" podUID="e5944af0-5719-461a-b5bc-0c05023e086d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 20:50:33.517208 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:33.516844    5071 generic.go:356] "Generic (PLEG): container finished" podID="da871b53-aae6-4363-95bb-78bca2b3d48f" containerID="ce46d77cd42b8fdeb83e37a9b9001432751063d511835f65e58f6bd905bc0cf8" exitCode=0
Oct 30 20:50:33.517208 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:33.516873    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc0e9f64-d2s85" event={"ID":"da871b53-aae6-4363-95bb-78bca2b3d48f","Type":"ContainerDied","Data":"ce46d77cd42b8fdeb83e37a9b9001432751063d511835f65e58f6bd905bc0cf8"}
Oct 30 20:50:33.977991 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:33.977956    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:50:34.076258 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:34.076216    5071 patch_prober.go:29] interesting pod/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 20:50:34.076568 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:34.076532    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c" podUID="04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 20:50:34.307028 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:34.306999    5071 patch_prober.go:29] interesting pod/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 20:50:34.307197 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:34.307048    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7" podUID="e5944af0-5719-461a-b5bc-0c05023e086d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 20:50:34.985903 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:34.982434    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:50:34.988357 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:34.988317    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc0e9f64-d2s85"
Oct 30 20:50:34.998917 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:34.998673    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:35.073240 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:35.073202    5071 patch_prober.go:29] interesting pod/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 20:50:35.073555 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:35.073523    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c" podUID="04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 20:50:35.101554 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:35.101515    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da871b53-aae6-4363-95bb-78bca2b3d48f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da871b53-aae6-4363-95bb-78bca2b3d48f-maas-api-openshift-ca\") pod \"da871b53-aae6-4363-95bb-78bca2b3d48f\" (UID: \"da871b53-aae6-4363-95bb-78bca2b3d48f\") "
Oct 30 20:50:35.101754 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:35.101582    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da871b53-aae6-4363-95bb-78bca2b3d48f-kube-api-access-hhx7s\" (UniqueName: \"kubernetes.io/projected/da871b53-aae6-4363-95bb-78bca2b3d48f-kube-api-access-hhx7s\") pod \"da871b53-aae6-4363-95bb-78bca2b3d48f\" (UID: \"da871b53-aae6-4363-95bb-78bca2b3d48f\") "
Oct 30 20:50:35.101754 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:35.101662    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da871b53-aae6-4363-95bb-78bca2b3d48f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da871b53-aae6-4363-95bb-78bca2b3d48f-maas-api-xks-ca\") pod \"da871b53-aae6-4363-95bb-78bca2b3d48f\" (UID: \"da871b53-aae6-4363-95bb-78bca2b3d48f\") "
Oct 30 20:50:35.102224 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:35.101919    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da871b53-aae6-4363-95bb-78bca2b3d48f-maas-api-openshift-ca" pod "da871b53-aae6-4363-95bb-78bca2b3d48f" (UID: "da871b53-aae6-4363-95bb-78bca2b3d48f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:50:35.102508 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:35.102457    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da871b53-aae6-4363-95bb-78bca2b3d48f-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:50:35.104964 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:35.104937    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da871b53-aae6-4363-95bb-78bca2b3d48f-maas-api-xks-ca" pod "da871b53-aae6-4363-95bb-78bca2b3d48f" (UID: "da871b53-aae6-4363-95bb-78bca2b3d48f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:50:35.104964 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:35.104947    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da871b53-aae6-4363-95bb-78bca2b3d48f-kube-api-access-hhx7s" pod "da871b53-aae6-4363-95bb-78bca2b3d48f" (UID: "da871b53-aae6-4363-95bb-78bca2b3d48f"). InnerVolumeSpecName "kube-api-access-hhx7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:50:35.208339 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:35.208280    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhx7s\" (UniqueName: \"kubernetes.io/projected/da871b53-aae6-4363-95bb-78bca2b3d48f-kube-api-access-hhx7s\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:50:35.208568 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:35.208542    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da871b53-aae6-4363-95bb-78bca2b3d48f-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:50:35.306842 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:35.306570    5071 patch_prober.go:29] interesting pod/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 20:50:35.306842 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:35.306627    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7" podUID="e5944af0-5719-461a-b5bc-0c05023e086d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 20:50:35.595908 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:35.592685    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc0e9f64-d2s85" event={"ID":"da871b53-aae6-4363-95bb-78bca2b3d48f","Type":"ContainerDied","Data":"943a0736748247231ab05c9768b8945ea73f3e8a14e2d44f969855a70b552a08"}
Oct 30 20:50:35.595908 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:35.592752    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="943a0736748247231ab05c9768b8945ea73f3e8a14e2d44f969855a70b552a08"
Oct 30 20:50:35.595908 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:35.592875    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc0e9f64-d2s85"
Oct 30 20:50:36.077899 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:36.077289    5071 patch_prober.go:29] interesting pod/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 20:50:36.077899 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:36.077353    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c" podUID="04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 20:50:36.308850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:36.308210    5071 patch_prober.go:29] interesting pod/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 20:50:36.308850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:36.308271    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7" podUID="e5944af0-5719-461a-b5bc-0c05023e086d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 20:50:37.077596 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:37.077564    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:50:37.078580 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:37.078558    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:50:37.093564 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:37.093528    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:37.311298 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:37.311270    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:50:37.312340 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:37.312321    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:50:37.460901 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:37.457159    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:38.223258 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:38.223226    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd92310f-n5pd4"]
Oct 30 20:50:38.224070 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:38.224053    5071 cpu_manager.go:414] "containerMap: removing container" podUID="da871b53-aae6-4363-95bb-78bca2b3d48f" containerName="revoke-keys"
Oct 30 20:50:38.224070 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:38.224070    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da871b53-aae6-4363-95bb-78bca2b3d48f" containerName="revoke-keys"
Oct 30 20:50:38.224206 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:38.224197    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="da871b53-aae6-4363-95bb-78bca2b3d48f" containerName="revoke-keys"
Oct 30 20:50:38.227264 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:38.227248    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd92310f-n5pd4"
Oct 30 20:50:38.285891 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:38.285860    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:50:38.293180 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:38.293149    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd92310f-n5pd4"]
Oct 30 20:50:38.307614 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:38.307580    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/864ae5b2-2748-4a9e-9ab9-ec3adca7f91a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-bd92310f-n5pd4\" (UID: \"864ae5b2-2748-4a9e-9ab9-ec3adca7f91a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd92310f-n5pd4"
Oct 30 20:50:38.307799 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:38.307781    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/864ae5b2-2748-4a9e-9ab9-ec3adca7f91a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-bd92310f-n5pd4\" (UID: \"864ae5b2-2748-4a9e-9ab9-ec3adca7f91a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd92310f-n5pd4"
Oct 30 20:50:38.307925 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:38.307907    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h867w\" (UniqueName: \"kubernetes.io/projected/864ae5b2-2748-4a9e-9ab9-ec3adca7f91a-kube-api-access-h867w\") pod \"maas-api-revoke-keys-e2e-derive-bd92310f-n5pd4\" (UID: \"864ae5b2-2748-4a9e-9ab9-ec3adca7f91a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd92310f-n5pd4"
Oct 30 20:50:38.333902 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:38.333873    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 20:50:38.409670 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:38.409635    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/864ae5b2-2748-4a9e-9ab9-ec3adca7f91a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-bd92310f-n5pd4\" (UID: \"864ae5b2-2748-4a9e-9ab9-ec3adca7f91a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd92310f-n5pd4"
Oct 30 20:50:38.409881 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:38.409730    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h867w\" (UniqueName: \"kubernetes.io/projected/864ae5b2-2748-4a9e-9ab9-ec3adca7f91a-kube-api-access-h867w\") pod \"maas-api-revoke-keys-e2e-derive-bd92310f-n5pd4\" (UID: \"864ae5b2-2748-4a9e-9ab9-ec3adca7f91a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd92310f-n5pd4"
Oct 30 20:50:38.409881 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:38.409873    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/864ae5b2-2748-4a9e-9ab9-ec3adca7f91a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-bd92310f-n5pd4\" (UID: \"864ae5b2-2748-4a9e-9ab9-ec3adca7f91a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd92310f-n5pd4"
Oct 30 20:50:38.410527 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:38.410507    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/864ae5b2-2748-4a9e-9ab9-ec3adca7f91a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-bd92310f-n5pd4\" (UID: \"864ae5b2-2748-4a9e-9ab9-ec3adca7f91a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd92310f-n5pd4"
Oct 30 20:50:38.412226 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:38.412207    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/864ae5b2-2748-4a9e-9ab9-ec3adca7f91a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-bd92310f-n5pd4\" (UID: \"864ae5b2-2748-4a9e-9ab9-ec3adca7f91a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd92310f-n5pd4"
Oct 30 20:50:38.469700 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:38.469667    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h867w\" (UniqueName: \"kubernetes.io/projected/864ae5b2-2748-4a9e-9ab9-ec3adca7f91a-kube-api-access-h867w\") pod \"maas-api-revoke-keys-e2e-derive-bd92310f-n5pd4\" (UID: \"864ae5b2-2748-4a9e-9ab9-ec3adca7f91a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd92310f-n5pd4"
Oct 30 20:50:38.542232 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:38.542199    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd92310f-n5pd4"
Oct 30 20:50:38.934288 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:50:38.934253    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod864ae5b2_2748_4a9e_9ab9_ec3adca7f91a.slice/crio-d2e6943d8d33fa099bdcb4f25f41890fada12807eee8ce76aa07757d7aad554f WatchSource:0}: Error finding container d2e6943d8d33fa099bdcb4f25f41890fada12807eee8ce76aa07757d7aad554f: Status 404 returned error can't find the container with id d2e6943d8d33fa099bdcb4f25f41890fada12807eee8ce76aa07757d7aad554f
Oct 30 20:50:38.935553 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:38.935531    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd92310f-n5pd4"]
Oct 30 20:50:39.693559 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:39.693521    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd92310f-n5pd4" event={"ID":"864ae5b2-2748-4a9e-9ab9-ec3adca7f91a","Type":"ContainerStarted","Data":"0c99bcd96c6e941f2e680d9600249b536f2ad09013b8ecfd6d5d392a092f267a"}
Oct 30 20:50:39.693993 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:39.693579    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd92310f-n5pd4" event={"ID":"864ae5b2-2748-4a9e-9ab9-ec3adca7f91a","Type":"ContainerStarted","Data":"d2e6943d8d33fa099bdcb4f25f41890fada12807eee8ce76aa07757d7aad554f"}
Oct 30 20:50:39.811760 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:39.811702    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd92310f-n5pd4" podStartSLOduration=1.8116872819999998 podStartE2EDuration="1.811687282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:39.810072977 +0000 UTC m=+1451.727094514" watchObservedRunningTime="2026-10-30 20:50:39.811687282 +0000 UTC m=+1451.728708818"
Oct 30 20:50:40.706139 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:40.706091    5071 generic.go:356] "Generic (PLEG): container finished" podID="864ae5b2-2748-4a9e-9ab9-ec3adca7f91a" containerID="0c99bcd96c6e941f2e680d9600249b536f2ad09013b8ecfd6d5d392a092f267a" exitCode=0
Oct 30 20:50:40.706551 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:40.706223    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd92310f-n5pd4" event={"ID":"864ae5b2-2748-4a9e-9ab9-ec3adca7f91a","Type":"ContainerDied","Data":"0c99bcd96c6e941f2e680d9600249b536f2ad09013b8ecfd6d5d392a092f267a"}
Oct 30 20:50:41.941010 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:41.940979    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd92310f-n5pd4"
Oct 30 20:50:41.975252 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:41.975164    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/864ae5b2-2748-4a9e-9ab9-ec3adca7f91a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/864ae5b2-2748-4a9e-9ab9-ec3adca7f91a-maas-api-openshift-ca\") pod \"864ae5b2-2748-4a9e-9ab9-ec3adca7f91a\" (UID: \"864ae5b2-2748-4a9e-9ab9-ec3adca7f91a\") "
Oct 30 20:50:41.975433 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:41.975411    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/864ae5b2-2748-4a9e-9ab9-ec3adca7f91a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/864ae5b2-2748-4a9e-9ab9-ec3adca7f91a-maas-api-xks-ca\") pod \"864ae5b2-2748-4a9e-9ab9-ec3adca7f91a\" (UID: \"864ae5b2-2748-4a9e-9ab9-ec3adca7f91a\") "
Oct 30 20:50:41.975541 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:41.975515    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/864ae5b2-2748-4a9e-9ab9-ec3adca7f91a-maas-api-openshift-ca" pod "864ae5b2-2748-4a9e-9ab9-ec3adca7f91a" (UID: "864ae5b2-2748-4a9e-9ab9-ec3adca7f91a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:50:41.975594 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:41.975535    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/864ae5b2-2748-4a9e-9ab9-ec3adca7f91a-kube-api-access-h867w\" (UniqueName: \"kubernetes.io/projected/864ae5b2-2748-4a9e-9ab9-ec3adca7f91a-kube-api-access-h867w\") pod \"864ae5b2-2748-4a9e-9ab9-ec3adca7f91a\" (UID: \"864ae5b2-2748-4a9e-9ab9-ec3adca7f91a\") "
Oct 30 20:50:41.976396 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:41.976370    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/864ae5b2-2748-4a9e-9ab9-ec3adca7f91a-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:50:41.977888 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:41.977857    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/864ae5b2-2748-4a9e-9ab9-ec3adca7f91a-kube-api-access-h867w" pod "864ae5b2-2748-4a9e-9ab9-ec3adca7f91a" (UID: "864ae5b2-2748-4a9e-9ab9-ec3adca7f91a"). InnerVolumeSpecName "kube-api-access-h867w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:50:41.978255 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:41.978234    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/864ae5b2-2748-4a9e-9ab9-ec3adca7f91a-maas-api-xks-ca" pod "864ae5b2-2748-4a9e-9ab9-ec3adca7f91a" (UID: "864ae5b2-2748-4a9e-9ab9-ec3adca7f91a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:50:42.128983 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:42.096253    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/864ae5b2-2748-4a9e-9ab9-ec3adca7f91a-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:50:42.128983 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:42.128955    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h867w\" (UniqueName: \"kubernetes.io/projected/864ae5b2-2748-4a9e-9ab9-ec3adca7f91a-kube-api-access-h867w\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:50:42.523072 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:42.523031    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-s6hth container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:50:42.524049 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:42.524008    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth" podUID="ecd40162-84fb-4838-ae3b-bafa49f9cdb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:50:42.727188 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:42.727159    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd92310f-n5pd4"
Oct 30 20:50:42.727188 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:42.727187    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd92310f-n5pd4" event={"ID":"864ae5b2-2748-4a9e-9ab9-ec3adca7f91a","Type":"ContainerDied","Data":"d2e6943d8d33fa099bdcb4f25f41890fada12807eee8ce76aa07757d7aad554f"}
Oct 30 20:50:42.727412 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:42.727235    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2e6943d8d33fa099bdcb4f25f41890fada12807eee8ce76aa07757d7aad554f"
Oct 30 20:50:43.062513 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:43.062469    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31e7414a-ndg6b"]
Oct 30 20:50:43.075782 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:43.075750    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31e7414a-ndg6b"]
Oct 30 20:50:43.476913 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:43.473026    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:43.883768 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:43.883735    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9782c31-vxgfp"]
Oct 30 20:50:43.884553 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:43.884538    5071 cpu_manager.go:414] "containerMap: removing container" podUID="864ae5b2-2748-4a9e-9ab9-ec3adca7f91a" containerName="revoke-keys"
Oct 30 20:50:43.884600 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:43.884555    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="864ae5b2-2748-4a9e-9ab9-ec3adca7f91a" containerName="revoke-keys"
Oct 30 20:50:43.884702 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:43.884692    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="864ae5b2-2748-4a9e-9ab9-ec3adca7f91a" containerName="revoke-keys"
Oct 30 20:50:43.887636 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:43.887618    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9782c31-vxgfp"
Oct 30 20:50:43.942941 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:43.942915    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:50:43.943131 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:43.943115    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 20:50:44.001994 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:44.001962    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9782c31-vxgfp"]
Oct 30 20:50:44.039549 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:44.039513    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7n69\" (UniqueName: \"kubernetes.io/projected/559aa737-49ae-4c48-aff0-22a31eaf9698-kube-api-access-c7n69\") pod \"maas-api-revoke-keys-e2e-ait-f9782c31-vxgfp\" (UID: \"559aa737-49ae-4c48-aff0-22a31eaf9698\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9782c31-vxgfp"
Oct 30 20:50:44.039723 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:44.039638    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/559aa737-49ae-4c48-aff0-22a31eaf9698-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9782c31-vxgfp\" (UID: \"559aa737-49ae-4c48-aff0-22a31eaf9698\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9782c31-vxgfp"
Oct 30 20:50:44.039766 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:44.039741    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/559aa737-49ae-4c48-aff0-22a31eaf9698-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9782c31-vxgfp\" (UID: \"559aa737-49ae-4c48-aff0-22a31eaf9698\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9782c31-vxgfp"
Oct 30 20:50:44.141788 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:44.141709    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7n69\" (UniqueName: \"kubernetes.io/projected/559aa737-49ae-4c48-aff0-22a31eaf9698-kube-api-access-c7n69\") pod \"maas-api-revoke-keys-e2e-ait-f9782c31-vxgfp\" (UID: \"559aa737-49ae-4c48-aff0-22a31eaf9698\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9782c31-vxgfp"
Oct 30 20:50:44.141788 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:44.141782    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/559aa737-49ae-4c48-aff0-22a31eaf9698-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9782c31-vxgfp\" (UID: \"559aa737-49ae-4c48-aff0-22a31eaf9698\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9782c31-vxgfp"
Oct 30 20:50:44.142288 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:44.141916    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/559aa737-49ae-4c48-aff0-22a31eaf9698-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9782c31-vxgfp\" (UID: \"559aa737-49ae-4c48-aff0-22a31eaf9698\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9782c31-vxgfp"
Oct 30 20:50:44.142776 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:44.142756    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/559aa737-49ae-4c48-aff0-22a31eaf9698-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9782c31-vxgfp\" (UID: \"559aa737-49ae-4c48-aff0-22a31eaf9698\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9782c31-vxgfp"
Oct 30 20:50:44.144446 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:44.144428    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/559aa737-49ae-4c48-aff0-22a31eaf9698-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9782c31-vxgfp\" (UID: \"559aa737-49ae-4c48-aff0-22a31eaf9698\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9782c31-vxgfp"
Oct 30 20:50:44.200707 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:44.200678    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7n69\" (UniqueName: \"kubernetes.io/projected/559aa737-49ae-4c48-aff0-22a31eaf9698-kube-api-access-c7n69\") pod \"maas-api-revoke-keys-e2e-ait-f9782c31-vxgfp\" (UID: \"559aa737-49ae-4c48-aff0-22a31eaf9698\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9782c31-vxgfp"
Oct 30 20:50:44.200952 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:44.200935    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9782c31-vxgfp"
Oct 30 20:50:44.208940 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:44.208912    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32a4c593-1f5c-4060-a418-f216a4bc105f" path="/var/lib/kubelet/pods/32a4c593-1f5c-4060-a418-f216a4bc105f/volumes"
Oct 30 20:50:44.379088 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:44.379055    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:44.585707 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:50:44.585659    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod559aa737_49ae_4c48_aff0_22a31eaf9698.slice/crio-e8d8c1723a822232e0c6bcf7f2e14bcb5f57a6c340f7a1e92e8662751c4fdec4 WatchSource:0}: Error finding container e8d8c1723a822232e0c6bcf7f2e14bcb5f57a6c340f7a1e92e8662751c4fdec4: Status 404 returned error can't find the container with id e8d8c1723a822232e0c6bcf7f2e14bcb5f57a6c340f7a1e92e8662751c4fdec4
Oct 30 20:50:44.587624 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:44.587523    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9782c31-vxgfp"]
Oct 30 20:50:44.752119 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:44.752089    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9782c31-vxgfp" event={"ID":"559aa737-49ae-4c48-aff0-22a31eaf9698","Type":"ContainerStarted","Data":"e8d8c1723a822232e0c6bcf7f2e14bcb5f57a6c340f7a1e92e8662751c4fdec4"}
Oct 30 20:50:44.892362 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:44.892277    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-dc0e9f64-66454765d-kn2vl"]
Oct 30 20:50:45.290122 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:45.290090    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:45.546921 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:45.542561    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:45.762679 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:45.762635    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9782c31-vxgfp" event={"ID":"559aa737-49ae-4c48-aff0-22a31eaf9698","Type":"ContainerStarted","Data":"080315184c09283122dcfc85e137f020b9ec79c72735e4f8653ad404796ae5b1"}
Oct 30 20:50:45.881710 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:45.881645    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9782c31-vxgfp" podStartSLOduration=2.881624382 podStartE2EDuration="2.881624382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:45.877781325 +0000 UTC m=+1457.794802894" watchObservedRunningTime="2026-10-30 20:50:45.881624382 +0000 UTC m=+1457.798645919"
Oct 30 20:50:46.885809 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:46.884221    5071 generic.go:356] "Generic (PLEG): container finished" podID="559aa737-49ae-4c48-aff0-22a31eaf9698" containerID="080315184c09283122dcfc85e137f020b9ec79c72735e4f8653ad404796ae5b1" exitCode=0
Oct 30 20:50:46.885809 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:46.884535    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9782c31-vxgfp" event={"ID":"559aa737-49ae-4c48-aff0-22a31eaf9698","Type":"ContainerDied","Data":"080315184c09283122dcfc85e137f020b9ec79c72735e4f8653ad404796ae5b1"}
Oct 30 20:50:46.896859 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:50:46.893053    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:50:48.060150 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:48.060120    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9782c31-vxgfp"
Oct 30 20:50:48.145379 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:48.145283    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/559aa737-49ae-4c48-aff0-22a31eaf9698-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/559aa737-49ae-4c48-aff0-22a31eaf9698-maas-api-xks-ca\") pod \"559aa737-49ae-4c48-aff0-22a31eaf9698\" (UID: \"559aa737-49ae-4c48-aff0-22a31eaf9698\") "
Oct 30 20:50:48.145557 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:48.145515    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/559aa737-49ae-4c48-aff0-22a31eaf9698-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/559aa737-49ae-4c48-aff0-22a31eaf9698-maas-api-openshift-ca\") pod \"559aa737-49ae-4c48-aff0-22a31eaf9698\" (UID: \"559aa737-49ae-4c48-aff0-22a31eaf9698\") "
Oct 30 20:50:48.145626 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:48.145598    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/559aa737-49ae-4c48-aff0-22a31eaf9698-kube-api-access-c7n69\" (UniqueName: \"kubernetes.io/projected/559aa737-49ae-4c48-aff0-22a31eaf9698-kube-api-access-c7n69\") pod \"559aa737-49ae-4c48-aff0-22a31eaf9698\" (UID: \"559aa737-49ae-4c48-aff0-22a31eaf9698\") "
Oct 30 20:50:48.145895 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:48.145872    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/559aa737-49ae-4c48-aff0-22a31eaf9698-maas-api-openshift-ca" pod "559aa737-49ae-4c48-aff0-22a31eaf9698" (UID: "559aa737-49ae-4c48-aff0-22a31eaf9698"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:50:48.146298 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:48.146233    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/559aa737-49ae-4c48-aff0-22a31eaf9698-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:50:48.147657 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:48.147634    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/559aa737-49ae-4c48-aff0-22a31eaf9698-maas-api-xks-ca" pod "559aa737-49ae-4c48-aff0-22a31eaf9698" (UID: "559aa737-49ae-4c48-aff0-22a31eaf9698"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:50:48.147953 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:48.147929    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/559aa737-49ae-4c48-aff0-22a31eaf9698-kube-api-access-c7n69" pod "559aa737-49ae-4c48-aff0-22a31eaf9698" (UID: "559aa737-49ae-4c48-aff0-22a31eaf9698"). InnerVolumeSpecName "kube-api-access-c7n69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:50:48.248082 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:48.248050    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/559aa737-49ae-4c48-aff0-22a31eaf9698-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:50:48.248082 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:48.248081    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7n69\" (UniqueName: \"kubernetes.io/projected/559aa737-49ae-4c48-aff0-22a31eaf9698-kube-api-access-c7n69\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:50:48.852131 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:48.852098    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bd92310f-7f56b44dcd-txxfn"]
Oct 30 20:50:48.904095 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:48.904064    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9782c31-vxgfp"
Oct 30 20:50:48.904281 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:48.904103    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9782c31-vxgfp" event={"ID":"559aa737-49ae-4c48-aff0-22a31eaf9698","Type":"ContainerDied","Data":"e8d8c1723a822232e0c6bcf7f2e14bcb5f57a6c340f7a1e92e8662751c4fdec4"}
Oct 30 20:50:48.904281 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:48.904151    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8d8c1723a822232e0c6bcf7f2e14bcb5f57a6c340f7a1e92e8662751c4fdec4"
Oct 30 20:50:48.984994 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:48.984964    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bd92310f-d9b4b64d7-pbvp7"]
Oct 30 20:50:49.242326 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:49.242231    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:49.494900 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:49.473486    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:49.791140 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:49.791089    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:49.895984 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:49.895898    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-dc0e9f64-66454765d-kn2vl" podUID="aac6152c-a8ea-4988-996a-e132f9c29c18" containerName="payload-processing" containerID="cri-o://bac6abe423f3aba15db82c8f82eed035a79508ad42827e1a54eca96b5723cce6" gracePeriod=25
Oct 30 20:50:49.937561 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:49.937533    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78ca07a3-74lzx"]
Oct 30 20:50:49.939198 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:49.939151    5071 cpu_manager.go:414] "containerMap: removing container" podUID="559aa737-49ae-4c48-aff0-22a31eaf9698" containerName="revoke-keys"
Oct 30 20:50:49.939198 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:49.939178    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="559aa737-49ae-4c48-aff0-22a31eaf9698" containerName="revoke-keys"
Oct 30 20:50:49.939585 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:49.939481    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="559aa737-49ae-4c48-aff0-22a31eaf9698" containerName="revoke-keys"
Oct 30 20:50:49.943326 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:49.943303    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78ca07a3-74lzx"
Oct 30 20:50:49.965117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:49.965081    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ab1657-d608-474f-b6ed-23ccb44198a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-78ca07a3-74lzx\" (UID: \"43ab1657-d608-474f-b6ed-23ccb44198a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78ca07a3-74lzx"
Oct 30 20:50:49.965272 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:49.965144    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zr2p\" (UniqueName: \"kubernetes.io/projected/43ab1657-d608-474f-b6ed-23ccb44198a7-kube-api-access-9zr2p\") pod \"maas-api-revoke-keys-e2e-ait-78ca07a3-74lzx\" (UID: \"43ab1657-d608-474f-b6ed-23ccb44198a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78ca07a3-74lzx"
Oct 30 20:50:49.965336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:49.965311    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ab1657-d608-474f-b6ed-23ccb44198a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-78ca07a3-74lzx\" (UID: \"43ab1657-d608-474f-b6ed-23ccb44198a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78ca07a3-74lzx"
Oct 30 20:50:50.001311 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:50.000991    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:50:50.001781 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:50.001744    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 20:50:50.007117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:50.006785    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78ca07a3-74lzx"]
Oct 30 20:50:50.068160 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:50.068077    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ab1657-d608-474f-b6ed-23ccb44198a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-78ca07a3-74lzx\" (UID: \"43ab1657-d608-474f-b6ed-23ccb44198a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78ca07a3-74lzx"
Oct 30 20:50:50.068160 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:50.068149    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zr2p\" (UniqueName: \"kubernetes.io/projected/43ab1657-d608-474f-b6ed-23ccb44198a7-kube-api-access-9zr2p\") pod \"maas-api-revoke-keys-e2e-ait-78ca07a3-74lzx\" (UID: \"43ab1657-d608-474f-b6ed-23ccb44198a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78ca07a3-74lzx"
Oct 30 20:50:50.068350 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:50.068242    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ab1657-d608-474f-b6ed-23ccb44198a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-78ca07a3-74lzx\" (UID: \"43ab1657-d608-474f-b6ed-23ccb44198a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78ca07a3-74lzx"
Oct 30 20:50:50.069090 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:50.069065    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ab1657-d608-474f-b6ed-23ccb44198a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-78ca07a3-74lzx\" (UID: \"43ab1657-d608-474f-b6ed-23ccb44198a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78ca07a3-74lzx"
Oct 30 20:50:50.073868 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:50.072527    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ab1657-d608-474f-b6ed-23ccb44198a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-78ca07a3-74lzx\" (UID: \"43ab1657-d608-474f-b6ed-23ccb44198a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78ca07a3-74lzx"
Oct 30 20:50:50.134847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:50.132101    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zr2p\" (UniqueName: \"kubernetes.io/projected/43ab1657-d608-474f-b6ed-23ccb44198a7-kube-api-access-9zr2p\") pod \"maas-api-revoke-keys-e2e-ait-78ca07a3-74lzx\" (UID: \"43ab1657-d608-474f-b6ed-23ccb44198a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78ca07a3-74lzx"
Oct 30 20:50:50.289048 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:50.289017    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-dc0e9f64-66454765d-kn2vl"
Oct 30 20:50:50.318902 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:50.318745    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78ca07a3-74lzx"
Oct 30 20:50:50.386852 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:50.380071    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aac6152c-a8ea-4988-996a-e132f9c29c18-kube-api-access-27rgp\" (UniqueName: \"kubernetes.io/projected/aac6152c-a8ea-4988-996a-e132f9c29c18-kube-api-access-27rgp\") pod \"aac6152c-a8ea-4988-996a-e132f9c29c18\" (UID: \"aac6152c-a8ea-4988-996a-e132f9c29c18\") "
Oct 30 20:50:50.386852 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:50.380259    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aac6152c-a8ea-4988-996a-e132f9c29c18-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aac6152c-a8ea-4988-996a-e132f9c29c18-plugins-config-volume\") pod \"aac6152c-a8ea-4988-996a-e132f9c29c18\" (UID: \"aac6152c-a8ea-4988-996a-e132f9c29c18\") "
Oct 30 20:50:50.395847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:50.392474    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aac6152c-a8ea-4988-996a-e132f9c29c18-plugins-config-volume" pod "aac6152c-a8ea-4988-996a-e132f9c29c18" (UID: "aac6152c-a8ea-4988-996a-e132f9c29c18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:50:50.419848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:50.418777    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac6152c-a8ea-4988-996a-e132f9c29c18-kube-api-access-27rgp" pod "aac6152c-a8ea-4988-996a-e132f9c29c18" (UID: "aac6152c-a8ea-4988-996a-e132f9c29c18"). InnerVolumeSpecName "kube-api-access-27rgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:50:50.526094 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:50.525100    5071 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aac6152c-a8ea-4988-996a-e132f9c29c18-plugins-config-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:50:50.526094 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:50.525141    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27rgp\" (UniqueName: \"kubernetes.io/projected/aac6152c-a8ea-4988-996a-e132f9c29c18-kube-api-access-27rgp\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:50:51.051012 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:51.050979    5071 generic.go:356] "Generic (PLEG): container finished" podID="aac6152c-a8ea-4988-996a-e132f9c29c18" containerID="bac6abe423f3aba15db82c8f82eed035a79508ad42827e1a54eca96b5723cce6" exitCode=0
Oct 30 20:50:51.051161 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:51.051143    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-dc0e9f64-66454765d-kn2vl" event={"ID":"aac6152c-a8ea-4988-996a-e132f9c29c18","Type":"ContainerDied","Data":"bac6abe423f3aba15db82c8f82eed035a79508ad42827e1a54eca96b5723cce6"}
Oct 30 20:50:51.051218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:51.051201    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-dc0e9f64-66454765d-kn2vl" event={"ID":"aac6152c-a8ea-4988-996a-e132f9c29c18","Type":"ContainerDied","Data":"adae78ae8fdc1668422c61df7071ed5b8f6083a0e5cf8aada8f4d56e8c5670f7"}
Oct 30 20:50:51.051274 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:51.051233    5071 scope.go:122] "RemoveContainer" containerID="bac6abe423f3aba15db82c8f82eed035a79508ad42827e1a54eca96b5723cce6"
Oct 30 20:50:51.051409 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:51.051396    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-dc0e9f64-66454765d-kn2vl"
Oct 30 20:50:51.100552 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:51.100528    5071 scope.go:122] "RemoveContainer" containerID="bac6abe423f3aba15db82c8f82eed035a79508ad42827e1a54eca96b5723cce6"
Oct 30 20:50:51.101314 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:50:51.101234    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bac6abe423f3aba15db82c8f82eed035a79508ad42827e1a54eca96b5723cce6\": container with ID starting with bac6abe423f3aba15db82c8f82eed035a79508ad42827e1a54eca96b5723cce6 not found: ID does not exist" containerID="bac6abe423f3aba15db82c8f82eed035a79508ad42827e1a54eca96b5723cce6"
Oct 30 20:50:51.101314 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:51.101268    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bac6abe423f3aba15db82c8f82eed035a79508ad42827e1a54eca96b5723cce6"} err="failed to get container status \"bac6abe423f3aba15db82c8f82eed035a79508ad42827e1a54eca96b5723cce6\": rpc error: code = NotFound desc = could not find container \"bac6abe423f3aba15db82c8f82eed035a79508ad42827e1a54eca96b5723cce6\": container with ID starting with bac6abe423f3aba15db82c8f82eed035a79508ad42827e1a54eca96b5723cce6 not found: ID does not exist"
Oct 30 20:50:51.228241 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:51.228148    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-dc0e9f64-66454765d-kn2vl"]
Oct 30 20:50:51.234793 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:51.234766    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-dc0e9f64-66454765d-kn2vl"]
Oct 30 20:50:51.429308 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:50:51.429279    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ab1657_d608_474f_b6ed_23ccb44198a7.slice/crio-c6a4c9bf51bfcf4a5fc273aabd23ee07be239789355f906801b5ce26f0b08d62 WatchSource:0}: Error finding container c6a4c9bf51bfcf4a5fc273aabd23ee07be239789355f906801b5ce26f0b08d62: Status 404 returned error can't find the container with id c6a4c9bf51bfcf4a5fc273aabd23ee07be239789355f906801b5ce26f0b08d62
Oct 30 20:50:51.430212 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:51.430190    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78ca07a3-74lzx"]
Oct 30 20:50:52.063866 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:52.063804    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78ca07a3-74lzx" event={"ID":"43ab1657-d608-474f-b6ed-23ccb44198a7","Type":"ContainerStarted","Data":"69cf5477e64e502404b807865ee3ca1517321b8520f1c70579a3ead966cb550a"}
Oct 30 20:50:52.064019 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:52.063887    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78ca07a3-74lzx" event={"ID":"43ab1657-d608-474f-b6ed-23ccb44198a7","Type":"ContainerStarted","Data":"c6a4c9bf51bfcf4a5fc273aabd23ee07be239789355f906801b5ce26f0b08d62"}
Oct 30 20:50:52.185936 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:52.185883    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78ca07a3-74lzx" podStartSLOduration=3.1858679739999998 podStartE2EDuration="3.185867974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:52.181101531 +0000 UTC m=+1464.098123062" watchObservedRunningTime="2026-10-30 20:50:52.185867974 +0000 UTC m=+1464.102889507"
Oct 30 20:50:52.205498 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:52.205468    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aac6152c-a8ea-4988-996a-e132f9c29c18" path="/var/lib/kubelet/pods/aac6152c-a8ea-4988-996a-e132f9c29c18/volumes"
Oct 30 20:50:53.074362 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:53.074332    5071 generic.go:356] "Generic (PLEG): container finished" podID="43ab1657-d608-474f-b6ed-23ccb44198a7" containerID="69cf5477e64e502404b807865ee3ca1517321b8520f1c70579a3ead966cb550a" exitCode=0
Oct 30 20:50:53.074787 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:53.074422    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78ca07a3-74lzx" event={"ID":"43ab1657-d608-474f-b6ed-23ccb44198a7","Type":"ContainerDied","Data":"69cf5477e64e502404b807865ee3ca1517321b8520f1c70579a3ead966cb550a"}
Oct 30 20:50:53.853389 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:53.853336    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-bd92310f-7f56b44dcd-txxfn" podUID="eae68bfa-6442-4208-b363-71f7953d875e" containerName="payload-processing" containerID="cri-o://6d43c1fa71dbf17aa4426766b4bd51c0616d239806d37de10d0fa18ae00efa27" gracePeriod=25
Oct 30 20:50:53.985272 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:53.985238    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-bd92310f-d9b4b64d7-pbvp7" podUID="576450be-aec9-44a7-a91b-b2bd82342e57" containerName="payload-pre-processing" containerID="cri-o://40c636166e5358ce0f65d91b8c279afc9b5a229fa137e0732e78889cf81781d4" gracePeriod=25
Oct 30 20:50:54.103909 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:54.102235    5071 generic.go:356] "Generic (PLEG): container finished" podID="eae68bfa-6442-4208-b363-71f7953d875e" containerID="6d43c1fa71dbf17aa4426766b4bd51c0616d239806d37de10d0fa18ae00efa27" exitCode=0
Oct 30 20:50:54.103909 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:54.102320    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-bd92310f-7f56b44dcd-txxfn" event={"ID":"eae68bfa-6442-4208-b363-71f7953d875e","Type":"ContainerDied","Data":"6d43c1fa71dbf17aa4426766b4bd51c0616d239806d37de10d0fa18ae00efa27"}
Oct 30 20:50:54.213982 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:54.213012    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-bd92310f-7f56b44dcd-txxfn"
Oct 30 20:50:54.312031 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:54.312004    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78ca07a3-74lzx"
Oct 30 20:50:54.344880 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:54.331927    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eae68bfa-6442-4208-b363-71f7953d875e-kube-api-access-scxx5\" (UniqueName: \"kubernetes.io/projected/eae68bfa-6442-4208-b363-71f7953d875e-kube-api-access-scxx5\") pod \"eae68bfa-6442-4208-b363-71f7953d875e\" (UID: \"eae68bfa-6442-4208-b363-71f7953d875e\") "
Oct 30 20:50:54.344880 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:54.332245    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eae68bfa-6442-4208-b363-71f7953d875e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eae68bfa-6442-4208-b363-71f7953d875e-plugins-config-volume\") pod \"eae68bfa-6442-4208-b363-71f7953d875e\" (UID: \"eae68bfa-6442-4208-b363-71f7953d875e\") "
Oct 30 20:50:54.344880 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:54.338651    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eae68bfa-6442-4208-b363-71f7953d875e-plugins-config-volume" pod "eae68bfa-6442-4208-b363-71f7953d875e" (UID: "eae68bfa-6442-4208-b363-71f7953d875e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:50:54.344880 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:54.343004    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eae68bfa-6442-4208-b363-71f7953d875e-kube-api-access-scxx5" pod "eae68bfa-6442-4208-b363-71f7953d875e" (UID: "eae68bfa-6442-4208-b363-71f7953d875e"). InnerVolumeSpecName "kube-api-access-scxx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:50:54.344880 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:54.343365    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bd92310f-d9b4b64d7-pbvp7"
Oct 30 20:50:54.435516 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:54.435419    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43ab1657-d608-474f-b6ed-23ccb44198a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ab1657-d608-474f-b6ed-23ccb44198a7-maas-api-openshift-ca\") pod \"43ab1657-d608-474f-b6ed-23ccb44198a7\" (UID: \"43ab1657-d608-474f-b6ed-23ccb44198a7\") "
Oct 30 20:50:54.435689 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:54.435670    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/576450be-aec9-44a7-a91b-b2bd82342e57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/576450be-aec9-44a7-a91b-b2bd82342e57-plugins-config-volume\") pod \"576450be-aec9-44a7-a91b-b2bd82342e57\" (UID: \"576450be-aec9-44a7-a91b-b2bd82342e57\") "
Oct 30 20:50:54.435749 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:54.435717    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43ab1657-d608-474f-b6ed-23ccb44198a7-kube-api-access-9zr2p\" (UniqueName: \"kubernetes.io/projected/43ab1657-d608-474f-b6ed-23ccb44198a7-kube-api-access-9zr2p\") pod \"43ab1657-d608-474f-b6ed-23ccb44198a7\" (UID: \"43ab1657-d608-474f-b6ed-23ccb44198a7\") "
Oct 30 20:50:54.435813 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:54.435758    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ab1657-d608-474f-b6ed-23ccb44198a7-maas-api-openshift-ca" pod "43ab1657-d608-474f-b6ed-23ccb44198a7" (UID: "43ab1657-d608-474f-b6ed-23ccb44198a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:50:54.435813 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:54.435789    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43ab1657-d608-474f-b6ed-23ccb44198a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ab1657-d608-474f-b6ed-23ccb44198a7-maas-api-xks-ca\") pod \"43ab1657-d608-474f-b6ed-23ccb44198a7\" (UID: \"43ab1657-d608-474f-b6ed-23ccb44198a7\") "
Oct 30 20:50:54.435978 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:54.435912    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/576450be-aec9-44a7-a91b-b2bd82342e57-kube-api-access-954n4\" (UniqueName: \"kubernetes.io/projected/576450be-aec9-44a7-a91b-b2bd82342e57-kube-api-access-954n4\") pod \"576450be-aec9-44a7-a91b-b2bd82342e57\" (UID: \"576450be-aec9-44a7-a91b-b2bd82342e57\") "
Oct 30 20:50:54.436166 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:54.436139    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/576450be-aec9-44a7-a91b-b2bd82342e57-plugins-config-volume" pod "576450be-aec9-44a7-a91b-b2bd82342e57" (UID: "576450be-aec9-44a7-a91b-b2bd82342e57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:50:54.436960 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:54.436929    5071 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eae68bfa-6442-4208-b363-71f7953d875e-plugins-config-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:50:54.437049 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:54.436961    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ab1657-d608-474f-b6ed-23ccb44198a7-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:50:54.437049 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:54.436978    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scxx5\" (UniqueName: \"kubernetes.io/projected/eae68bfa-6442-4208-b363-71f7953d875e-kube-api-access-scxx5\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:50:54.437049 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:54.436993    5071 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/576450be-aec9-44a7-a91b-b2bd82342e57-plugins-config-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:50:54.438236 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:54.438210    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/576450be-aec9-44a7-a91b-b2bd82342e57-kube-api-access-954n4" pod "576450be-aec9-44a7-a91b-b2bd82342e57" (UID: "576450be-aec9-44a7-a91b-b2bd82342e57"). InnerVolumeSpecName "kube-api-access-954n4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:50:54.438331 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:54.438249    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43ab1657-d608-474f-b6ed-23ccb44198a7-maas-api-xks-ca" pod "43ab1657-d608-474f-b6ed-23ccb44198a7" (UID: "43ab1657-d608-474f-b6ed-23ccb44198a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:50:54.438415 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:54.438399    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43ab1657-d608-474f-b6ed-23ccb44198a7-kube-api-access-9zr2p" pod "43ab1657-d608-474f-b6ed-23ccb44198a7" (UID: "43ab1657-d608-474f-b6ed-23ccb44198a7"). InnerVolumeSpecName "kube-api-access-9zr2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:50:54.538634 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:54.538596    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zr2p\" (UniqueName: \"kubernetes.io/projected/43ab1657-d608-474f-b6ed-23ccb44198a7-kube-api-access-9zr2p\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:50:54.538792 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:54.538645    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ab1657-d608-474f-b6ed-23ccb44198a7-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:50:54.538792 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:54.538663    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-954n4\" (UniqueName: \"kubernetes.io/projected/576450be-aec9-44a7-a91b-b2bd82342e57-kube-api-access-954n4\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:50:55.111943 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:55.111901    5071 generic.go:356] "Generic (PLEG): container finished" podID="576450be-aec9-44a7-a91b-b2bd82342e57" containerID="40c636166e5358ce0f65d91b8c279afc9b5a229fa137e0732e78889cf81781d4" exitCode=0
Oct 30 20:50:55.112406 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:55.111977    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bd92310f-d9b4b64d7-pbvp7"
Oct 30 20:50:55.112406 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:55.111987    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bd92310f-d9b4b64d7-pbvp7" event={"ID":"576450be-aec9-44a7-a91b-b2bd82342e57","Type":"ContainerDied","Data":"40c636166e5358ce0f65d91b8c279afc9b5a229fa137e0732e78889cf81781d4"}
Oct 30 20:50:55.112406 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:55.112077    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bd92310f-d9b4b64d7-pbvp7" event={"ID":"576450be-aec9-44a7-a91b-b2bd82342e57","Type":"ContainerDied","Data":"2def2ddde70cfcb0dd341f0a5e74178d33f4fb4368c1dc8f184dfed821b2cafb"}
Oct 30 20:50:55.112406 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:55.112108    5071 scope.go:122] "RemoveContainer" containerID="40c636166e5358ce0f65d91b8c279afc9b5a229fa137e0732e78889cf81781d4"
Oct 30 20:50:55.113787 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:55.113769    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78ca07a3-74lzx"
Oct 30 20:50:55.114453 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:55.113872    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78ca07a3-74lzx" event={"ID":"43ab1657-d608-474f-b6ed-23ccb44198a7","Type":"ContainerDied","Data":"c6a4c9bf51bfcf4a5fc273aabd23ee07be239789355f906801b5ce26f0b08d62"}
Oct 30 20:50:55.114581 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:55.114478    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6a4c9bf51bfcf4a5fc273aabd23ee07be239789355f906801b5ce26f0b08d62"
Oct 30 20:50:55.115661 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:55.115637    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-bd92310f-7f56b44dcd-txxfn" event={"ID":"eae68bfa-6442-4208-b363-71f7953d875e","Type":"ContainerDied","Data":"ea8737b6b3d1cb30302d1fb3e999b31d9075e94851575b15ff56e5ce84a62822"}
Oct 30 20:50:55.115799 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:55.115672    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-bd92310f-7f56b44dcd-txxfn"
Oct 30 20:50:55.125073 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:55.125054    5071 scope.go:122] "RemoveContainer" containerID="40c636166e5358ce0f65d91b8c279afc9b5a229fa137e0732e78889cf81781d4"
Oct 30 20:50:55.125380 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:50:55.125352    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40c636166e5358ce0f65d91b8c279afc9b5a229fa137e0732e78889cf81781d4\": container with ID starting with 40c636166e5358ce0f65d91b8c279afc9b5a229fa137e0732e78889cf81781d4 not found: ID does not exist" containerID="40c636166e5358ce0f65d91b8c279afc9b5a229fa137e0732e78889cf81781d4"
Oct 30 20:50:55.125491 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:55.125389    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40c636166e5358ce0f65d91b8c279afc9b5a229fa137e0732e78889cf81781d4"} err="failed to get container status \"40c636166e5358ce0f65d91b8c279afc9b5a229fa137e0732e78889cf81781d4\": rpc error: code = NotFound desc = could not find container \"40c636166e5358ce0f65d91b8c279afc9b5a229fa137e0732e78889cf81781d4\": container with ID starting with 40c636166e5358ce0f65d91b8c279afc9b5a229fa137e0732e78889cf81781d4 not found: ID does not exist"
Oct 30 20:50:55.125491 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:55.125412    5071 scope.go:122] "RemoveContainer" containerID="6d43c1fa71dbf17aa4426766b4bd51c0616d239806d37de10d0fa18ae00efa27"
Oct 30 20:50:55.283870 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:55.283492    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:55.285487 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:55.285462    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bd92310f-d9b4b64d7-pbvp7"]
Oct 30 20:50:55.340632 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:55.337788    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bd92310f-d9b4b64d7-pbvp7"]
Oct 30 20:50:55.457604 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:55.457523    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bd92310f-7f56b44dcd-txxfn"]
Oct 30 20:50:55.463618 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:55.463592    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bd92310f-7f56b44dcd-txxfn"]
Oct 30 20:50:56.210855 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:56.210800    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="576450be-aec9-44a7-a91b-b2bd82342e57" path="/var/lib/kubelet/pods/576450be-aec9-44a7-a91b-b2bd82342e57/volumes"
Oct 30 20:50:56.211345 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:56.211328    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eae68bfa-6442-4208-b363-71f7953d875e" path="/var/lib/kubelet/pods/eae68bfa-6442-4208-b363-71f7953d875e/volumes"
Oct 30 20:50:56.475999 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:56.475919    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:50:59.054778 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:59.054746    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae05d1da-5jv5l"]
Oct 30 20:50:59.104187 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:50:59.104153    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae05d1da-5jv5l"]
Oct 30 20:51:00.204763 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:00.204730    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10967423-677a-4cfe-ab12-8cda6ebac21a" path="/var/lib/kubelet/pods/10967423-677a-4cfe-ab12-8cda6ebac21a/volumes"
Oct 30 20:51:01.100585 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:01.100358    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:01.500999 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:01.500914    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-78ca07a3-6f8759c9f8-xz5m9"]
Oct 30 20:51:01.631184 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:01.631150    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-78ca07a3-db58645f-ckkt9"]
Oct 30 20:51:01.909673 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:01.909639    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:02.151455 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:02.151411    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:05.545296 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:05.545264    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:51:06.147094 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.147060    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg"]
Oct 30 20:51:06.147862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.147841    5071 cpu_manager.go:414] "containerMap: removing container" podUID="43ab1657-d608-474f-b6ed-23ccb44198a7" containerName="revoke-keys"
Oct 30 20:51:06.147862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.147863    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43ab1657-d608-474f-b6ed-23ccb44198a7" containerName="revoke-keys"
Oct 30 20:51:06.147994 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.147881    5071 cpu_manager.go:414] "containerMap: removing container" podUID="aac6152c-a8ea-4988-996a-e132f9c29c18" containerName="payload-processing"
Oct 30 20:51:06.147994 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.147887    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aac6152c-a8ea-4988-996a-e132f9c29c18" containerName="payload-processing"
Oct 30 20:51:06.147994 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.147901    5071 cpu_manager.go:414] "containerMap: removing container" podUID="eae68bfa-6442-4208-b363-71f7953d875e" containerName="payload-processing"
Oct 30 20:51:06.147994 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.147906    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eae68bfa-6442-4208-b363-71f7953d875e" containerName="payload-processing"
Oct 30 20:51:06.147994 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.147933    5071 cpu_manager.go:414] "containerMap: removing container" podUID="576450be-aec9-44a7-a91b-b2bd82342e57" containerName="payload-pre-processing"
Oct 30 20:51:06.147994 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.147938    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="576450be-aec9-44a7-a91b-b2bd82342e57" containerName="payload-pre-processing"
Oct 30 20:51:06.148184 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.148050    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="43ab1657-d608-474f-b6ed-23ccb44198a7" containerName="revoke-keys"
Oct 30 20:51:06.148184 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.148060    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="aac6152c-a8ea-4988-996a-e132f9c29c18" containerName="payload-processing"
Oct 30 20:51:06.148184 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.148067    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="eae68bfa-6442-4208-b363-71f7953d875e" containerName="payload-processing"
Oct 30 20:51:06.148184 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.148078    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="576450be-aec9-44a7-a91b-b2bd82342e57" containerName="payload-pre-processing"
Oct 30 20:51:06.154064 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.154038    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg"
Oct 30 20:51:06.183445 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.183414    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be8fbd1-6cb8-4718-8cfe-a45894b5d33e-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cv4kg\" (UID: \"9be8fbd1-6cb8-4718-8cfe-a45894b5d33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg"
Oct 30 20:51:06.183579 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.183464    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t97fm\" (UniqueName: \"kubernetes.io/projected/9be8fbd1-6cb8-4718-8cfe-a45894b5d33e-kube-api-access-t97fm\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cv4kg\" (UID: \"9be8fbd1-6cb8-4718-8cfe-a45894b5d33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg"
Oct 30 20:51:06.183579 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.183519    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9be8fbd1-6cb8-4718-8cfe-a45894b5d33e-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cv4kg\" (UID: \"9be8fbd1-6cb8-4718-8cfe-a45894b5d33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg"
Oct 30 20:51:06.183658 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.183583    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9be8fbd1-6cb8-4718-8cfe-a45894b5d33e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cv4kg\" (UID: \"9be8fbd1-6cb8-4718-8cfe-a45894b5d33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg"
Oct 30 20:51:06.183658 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.183615    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9be8fbd1-6cb8-4718-8cfe-a45894b5d33e-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cv4kg\" (UID: \"9be8fbd1-6cb8-4718-8cfe-a45894b5d33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg"
Oct 30 20:51:06.183658 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.183652    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be8fbd1-6cb8-4718-8cfe-a45894b5d33e-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cv4kg\" (UID: \"9be8fbd1-6cb8-4718-8cfe-a45894b5d33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg"
Oct 30 20:51:06.183755 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.183670    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9be8fbd1-6cb8-4718-8cfe-a45894b5d33e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cv4kg\" (UID: \"9be8fbd1-6cb8-4718-8cfe-a45894b5d33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg"
Oct 30 20:51:06.183907 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.183889    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9be8fbd1-6cb8-4718-8cfe-a45894b5d33e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cv4kg\" (UID: \"9be8fbd1-6cb8-4718-8cfe-a45894b5d33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg"
Oct 30 20:51:06.183966 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.183931    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9be8fbd1-6cb8-4718-8cfe-a45894b5d33e-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cv4kg\" (UID: \"9be8fbd1-6cb8-4718-8cfe-a45894b5d33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg"
Oct 30 20:51:06.254270 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.254235    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg"]
Oct 30 20:51:06.285268 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.285231    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9be8fbd1-6cb8-4718-8cfe-a45894b5d33e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cv4kg\" (UID: \"9be8fbd1-6cb8-4718-8cfe-a45894b5d33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg"
Oct 30 20:51:06.285471 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.285288    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9be8fbd1-6cb8-4718-8cfe-a45894b5d33e-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cv4kg\" (UID: \"9be8fbd1-6cb8-4718-8cfe-a45894b5d33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg"
Oct 30 20:51:06.285471 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.285386    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be8fbd1-6cb8-4718-8cfe-a45894b5d33e-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cv4kg\" (UID: \"9be8fbd1-6cb8-4718-8cfe-a45894b5d33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg"
Oct 30 20:51:06.285471 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.285420    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t97fm\" (UniqueName: \"kubernetes.io/projected/9be8fbd1-6cb8-4718-8cfe-a45894b5d33e-kube-api-access-t97fm\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cv4kg\" (UID: \"9be8fbd1-6cb8-4718-8cfe-a45894b5d33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg"
Oct 30 20:51:06.285670 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.285498    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9be8fbd1-6cb8-4718-8cfe-a45894b5d33e-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cv4kg\" (UID: \"9be8fbd1-6cb8-4718-8cfe-a45894b5d33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg"
Oct 30 20:51:06.285670 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.285591    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9be8fbd1-6cb8-4718-8cfe-a45894b5d33e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cv4kg\" (UID: \"9be8fbd1-6cb8-4718-8cfe-a45894b5d33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg"
Oct 30 20:51:06.285670 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.285645    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9be8fbd1-6cb8-4718-8cfe-a45894b5d33e-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cv4kg\" (UID: \"9be8fbd1-6cb8-4718-8cfe-a45894b5d33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg"
Oct 30 20:51:06.285862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.285708    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be8fbd1-6cb8-4718-8cfe-a45894b5d33e-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cv4kg\" (UID: \"9be8fbd1-6cb8-4718-8cfe-a45894b5d33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg"
Oct 30 20:51:06.285862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.285714    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9be8fbd1-6cb8-4718-8cfe-a45894b5d33e-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cv4kg\" (UID: \"9be8fbd1-6cb8-4718-8cfe-a45894b5d33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg"
Oct 30 20:51:06.285862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.285732    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9be8fbd1-6cb8-4718-8cfe-a45894b5d33e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cv4kg\" (UID: \"9be8fbd1-6cb8-4718-8cfe-a45894b5d33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg"
Oct 30 20:51:06.285862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.285763    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be8fbd1-6cb8-4718-8cfe-a45894b5d33e-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cv4kg\" (UID: \"9be8fbd1-6cb8-4718-8cfe-a45894b5d33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg"
Oct 30 20:51:06.286150 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.285903    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9be8fbd1-6cb8-4718-8cfe-a45894b5d33e-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cv4kg\" (UID: \"9be8fbd1-6cb8-4718-8cfe-a45894b5d33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg"
Oct 30 20:51:06.286323 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.286301    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be8fbd1-6cb8-4718-8cfe-a45894b5d33e-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cv4kg\" (UID: \"9be8fbd1-6cb8-4718-8cfe-a45894b5d33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg"
Oct 30 20:51:06.286421 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.286362    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9be8fbd1-6cb8-4718-8cfe-a45894b5d33e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cv4kg\" (UID: \"9be8fbd1-6cb8-4718-8cfe-a45894b5d33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg"
Oct 30 20:51:06.287862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.287837    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9be8fbd1-6cb8-4718-8cfe-a45894b5d33e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cv4kg\" (UID: \"9be8fbd1-6cb8-4718-8cfe-a45894b5d33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg"
Oct 30 20:51:06.288337 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.288320    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9be8fbd1-6cb8-4718-8cfe-a45894b5d33e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cv4kg\" (UID: \"9be8fbd1-6cb8-4718-8cfe-a45894b5d33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg"
Oct 30 20:51:06.350123 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.350089    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9be8fbd1-6cb8-4718-8cfe-a45894b5d33e-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cv4kg\" (UID: \"9be8fbd1-6cb8-4718-8cfe-a45894b5d33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg"
Oct 30 20:51:06.350273 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.350233    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t97fm\" (UniqueName: \"kubernetes.io/projected/9be8fbd1-6cb8-4718-8cfe-a45894b5d33e-kube-api-access-t97fm\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cv4kg\" (UID: \"9be8fbd1-6cb8-4718-8cfe-a45894b5d33e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg"
Oct 30 20:51:06.470533 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.470450    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg"
Oct 30 20:51:06.488473 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.488432    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:06.501979 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.501938    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-78ca07a3-6f8759c9f8-xz5m9" podUID="52c6baf2-74c7-4ee6-baee-7357190687e5" containerName="payload-processing" containerID="cri-o://c9e1dcd5aacfcd07d0383f0cbad36ba31431d232235f8c0b5e40758cb81f59e5" gracePeriod=25
Oct 30 20:51:06.754624 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:06.674654    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-78ca07a3-db58645f-ckkt9" podUID="7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d" containerName="payload-pre-processing" containerID="cri-o://a407650503221dcf29bdc6c94eeb0364cffb72adffccf45ada787c9f8c922a15" gracePeriod=25
Oct 30 20:51:07.015068 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:51:07.013889    5071 logging.go:55] [core] [Channel #3112 SubChannel #3113]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.135:50052", ServerName: "10.132.0.135:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.135:50052: connect: connection refused"
Oct 30 20:51:07.015410 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:51:07.015386    5071 logging.go:55] [core] [Channel #3110 SubChannel #3111]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.134:50052", ServerName: "10.132.0.134:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.134:50052: connect: connection refused"
Oct 30 20:51:07.022998 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.022960    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg"]
Oct 30 20:51:07.043294 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:51:07.043226    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9be8fbd1_6cb8_4718_8cfe_a45894b5d33e.slice/crio-c20b1e5e6644f3ab127b08d15376006d1b4f105223554120bd758e580eb38e1f WatchSource:0}: Error finding container c20b1e5e6644f3ab127b08d15376006d1b4f105223554120bd758e580eb38e1f: Status 404 returned error can't find the container with id c20b1e5e6644f3ab127b08d15376006d1b4f105223554120bd758e580eb38e1f
Oct 30 20:51:07.047789 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.047014    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:51:07.047789 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.047115    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:51:07.047789 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.047160    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:51:07.263631 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.262425    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-78ca07a3-6f8759c9f8-xz5m9"
Oct 30 20:51:07.354592 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.354559    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52c6baf2-74c7-4ee6-baee-7357190687e5-kube-api-access-ph846\" (UniqueName: \"kubernetes.io/projected/52c6baf2-74c7-4ee6-baee-7357190687e5-kube-api-access-ph846\") pod \"52c6baf2-74c7-4ee6-baee-7357190687e5\" (UID: \"52c6baf2-74c7-4ee6-baee-7357190687e5\") "
Oct 30 20:51:07.354843 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.354804    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52c6baf2-74c7-4ee6-baee-7357190687e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52c6baf2-74c7-4ee6-baee-7357190687e5-plugins-config-volume\") pod \"52c6baf2-74c7-4ee6-baee-7357190687e5\" (UID: \"52c6baf2-74c7-4ee6-baee-7357190687e5\") "
Oct 30 20:51:07.356141 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.356105    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52c6baf2-74c7-4ee6-baee-7357190687e5-plugins-config-volume" pod "52c6baf2-74c7-4ee6-baee-7357190687e5" (UID: "52c6baf2-74c7-4ee6-baee-7357190687e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:51:07.358569 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.358547    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c6baf2-74c7-4ee6-baee-7357190687e5-kube-api-access-ph846" pod "52c6baf2-74c7-4ee6-baee-7357190687e5" (UID: "52c6baf2-74c7-4ee6-baee-7357190687e5"). InnerVolumeSpecName "kube-api-access-ph846". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:51:07.361329 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.361305    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-78ca07a3-db58645f-ckkt9"
Oct 30 20:51:07.412804 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.412767    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg" event={"ID":"9be8fbd1-6cb8-4718-8cfe-a45894b5d33e","Type":"ContainerStarted","Data":"8f7fb5e1b14c7fee4b519dea722168333f158b0e67fefb6d5a754d7b46bbfbc9"}
Oct 30 20:51:07.413036 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.412903    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg" event={"ID":"9be8fbd1-6cb8-4718-8cfe-a45894b5d33e","Type":"ContainerStarted","Data":"c20b1e5e6644f3ab127b08d15376006d1b4f105223554120bd758e580eb38e1f"}
Oct 30 20:51:07.415490 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.415464    5071 generic.go:356] "Generic (PLEG): container finished" podID="52c6baf2-74c7-4ee6-baee-7357190687e5" containerID="c9e1dcd5aacfcd07d0383f0cbad36ba31431d232235f8c0b5e40758cb81f59e5" exitCode=0
Oct 30 20:51:07.415668 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.415547    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-78ca07a3-6f8759c9f8-xz5m9" event={"ID":"52c6baf2-74c7-4ee6-baee-7357190687e5","Type":"ContainerDied","Data":"c9e1dcd5aacfcd07d0383f0cbad36ba31431d232235f8c0b5e40758cb81f59e5"}
Oct 30 20:51:07.415668 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.415586    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-78ca07a3-6f8759c9f8-xz5m9" event={"ID":"52c6baf2-74c7-4ee6-baee-7357190687e5","Type":"ContainerDied","Data":"645813d0dcd401b5fe0d0fa7671dc74dfd6bc8cd804549ed2009ad47431ad986"}
Oct 30 20:51:07.415668 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.415613    5071 scope.go:122] "RemoveContainer" containerID="c9e1dcd5aacfcd07d0383f0cbad36ba31431d232235f8c0b5e40758cb81f59e5"
Oct 30 20:51:07.415973 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.415741    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-78ca07a3-6f8759c9f8-xz5m9"
Oct 30 20:51:07.420720 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.420699    5071 generic.go:356] "Generic (PLEG): container finished" podID="7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d" containerID="a407650503221dcf29bdc6c94eeb0364cffb72adffccf45ada787c9f8c922a15" exitCode=0
Oct 30 20:51:07.421109 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.421073    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-78ca07a3-db58645f-ckkt9" event={"ID":"7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d","Type":"ContainerDied","Data":"a407650503221dcf29bdc6c94eeb0364cffb72adffccf45ada787c9f8c922a15"}
Oct 30 20:51:07.421289 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.421274    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-78ca07a3-db58645f-ckkt9" event={"ID":"7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d","Type":"ContainerDied","Data":"984e50dda07437b86d815185fcf04c04b02a45e21f37f19203ac6a16388fffaf"}
Oct 30 20:51:07.421526 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.421477    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-78ca07a3-db58645f-ckkt9"
Oct 30 20:51:07.449472 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.449446    5071 scope.go:122] "RemoveContainer" containerID="c9e1dcd5aacfcd07d0383f0cbad36ba31431d232235f8c0b5e40758cb81f59e5"
Oct 30 20:51:07.449930 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:51:07.449876    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9e1dcd5aacfcd07d0383f0cbad36ba31431d232235f8c0b5e40758cb81f59e5\": container with ID starting with c9e1dcd5aacfcd07d0383f0cbad36ba31431d232235f8c0b5e40758cb81f59e5 not found: ID does not exist" containerID="c9e1dcd5aacfcd07d0383f0cbad36ba31431d232235f8c0b5e40758cb81f59e5"
Oct 30 20:51:07.449930 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.449912    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9e1dcd5aacfcd07d0383f0cbad36ba31431d232235f8c0b5e40758cb81f59e5"} err="failed to get container status \"c9e1dcd5aacfcd07d0383f0cbad36ba31431d232235f8c0b5e40758cb81f59e5\": rpc error: code = NotFound desc = could not find container \"c9e1dcd5aacfcd07d0383f0cbad36ba31431d232235f8c0b5e40758cb81f59e5\": container with ID starting with c9e1dcd5aacfcd07d0383f0cbad36ba31431d232235f8c0b5e40758cb81f59e5 not found: ID does not exist"
Oct 30 20:51:07.450073 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.449936    5071 scope.go:122] "RemoveContainer" containerID="a407650503221dcf29bdc6c94eeb0364cffb72adffccf45ada787c9f8c922a15"
Oct 30 20:51:07.460163 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.457659    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d-plugins-config-volume\") pod \"7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d\" (UID: \"7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d\") "
Oct 30 20:51:07.460163 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.457967    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d-kube-api-access-jtw2h\" (UniqueName: \"kubernetes.io/projected/7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d-kube-api-access-jtw2h\") pod \"7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d\" (UID: \"7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d\") "
Oct 30 20:51:07.460163 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.458936    5071 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52c6baf2-74c7-4ee6-baee-7357190687e5-plugins-config-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:51:07.460163 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.458962    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph846\" (UniqueName: \"kubernetes.io/projected/52c6baf2-74c7-4ee6-baee-7357190687e5-kube-api-access-ph846\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:51:07.460163 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.459515    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d-plugins-config-volume" pod "7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d" (UID: "7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:51:07.468224 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.468201    5071 scope.go:122] "RemoveContainer" containerID="a407650503221dcf29bdc6c94eeb0364cffb72adffccf45ada787c9f8c922a15"
Oct 30 20:51:07.468675 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.468575    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d-kube-api-access-jtw2h" pod "7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d" (UID: "7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d"). InnerVolumeSpecName "kube-api-access-jtw2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:51:07.468783 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:51:07.468739    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a407650503221dcf29bdc6c94eeb0364cffb72adffccf45ada787c9f8c922a15\": container with ID starting with a407650503221dcf29bdc6c94eeb0364cffb72adffccf45ada787c9f8c922a15 not found: ID does not exist" containerID="a407650503221dcf29bdc6c94eeb0364cffb72adffccf45ada787c9f8c922a15"
Oct 30 20:51:07.468862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.468771    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a407650503221dcf29bdc6c94eeb0364cffb72adffccf45ada787c9f8c922a15"} err="failed to get container status \"a407650503221dcf29bdc6c94eeb0364cffb72adffccf45ada787c9f8c922a15\": rpc error: code = NotFound desc = could not find container \"a407650503221dcf29bdc6c94eeb0364cffb72adffccf45ada787c9f8c922a15\": container with ID starting with a407650503221dcf29bdc6c94eeb0364cffb72adffccf45ada787c9f8c922a15 not found: ID does not exist"
Oct 30 20:51:07.471123 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.471094    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg"
Oct 30 20:51:07.561651 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.561609    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtw2h\" (UniqueName: \"kubernetes.io/projected/7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d-kube-api-access-jtw2h\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:51:07.561651 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.561652    5071 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d-plugins-config-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:51:07.679855 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.679773    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-78ca07a3-6f8759c9f8-xz5m9"]
Oct 30 20:51:07.687082 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.687041    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-78ca07a3-6f8759c9f8-xz5m9"]
Oct 30 20:51:07.802312 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.802251    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg" podStartSLOduration=1.802236811 podStartE2EDuration="1.802236811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:07.79678307 +0000 UTC m=+1479.713804641" watchObservedRunningTime="2026-10-30 20:51:07.802236811 +0000 UTC m=+1479.719258347"
Oct 30 20:51:07.916325 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.916234    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-78ca07a3-db58645f-ckkt9"]
Oct 30 20:51:07.921514 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.921490    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-78ca07a3-db58645f-ckkt9"]
Oct 30 20:51:07.968655 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.968599    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ait-78ca07a3-db58645f-ckkt9" podUID="7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.134:50052\" within 1s: context deadline exceeded"
Oct 30 20:51:07.994218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:07.994175    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ait-78ca07a3-6f8759c9f8-xz5m9" podUID="52c6baf2-74c7-4ee6-baee-7357190687e5" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.135:50052\" within 1s: context deadline exceeded"
Oct 30 20:51:08.208486 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:08.208390    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52c6baf2-74c7-4ee6-baee-7357190687e5" path="/var/lib/kubelet/pods/52c6baf2-74c7-4ee6-baee-7357190687e5/volumes"
Oct 30 20:51:08.209055 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:08.209033    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d" path="/var/lib/kubelet/pods/7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d/volumes"
Oct 30 20:51:08.474901 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:08.472423    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cv4kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 20:51:08.474901 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:08.472510    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg" podUID="9be8fbd1-6cb8-4718-8cfe-a45894b5d33e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 20:51:09.482176 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:09.482141    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cv4kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 20:51:09.482781 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:09.482751    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg" podUID="9be8fbd1-6cb8-4718-8cfe-a45894b5d33e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 20:51:10.470736 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:10.470699    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cv4kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 20:51:10.470986 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:10.470750    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg" podUID="9be8fbd1-6cb8-4718-8cfe-a45894b5d33e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 20:51:11.471738 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:11.471704    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cv4kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 20:51:11.472281 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:11.471762    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg" podUID="9be8fbd1-6cb8-4718-8cfe-a45894b5d33e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 20:51:11.660139 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:11.658150    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:12.471783 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:12.471747    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cv4kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 20:51:12.472341 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:12.471812    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg" podUID="9be8fbd1-6cb8-4718-8cfe-a45894b5d33e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 20:51:13.471227 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:13.471185    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cv4kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 20:51:13.471449 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:13.471255    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg" podUID="9be8fbd1-6cb8-4718-8cfe-a45894b5d33e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 20:51:14.471725 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:14.471684    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cv4kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 20:51:14.472220 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:14.471755    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg" podUID="9be8fbd1-6cb8-4718-8cfe-a45894b5d33e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 20:51:15.927172 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:15.927141    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg"
Oct 30 20:51:15.927770 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:15.927731    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg"
Oct 30 20:51:15.928553 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:15.928530    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg"
Oct 30 20:51:16.681848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:16.680964    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:16.801859 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:51:16.794856    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:51:21.676052 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:21.676008    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:25.059537 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:25.059498    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36a4e29c-hhw62"]
Oct 30 20:51:25.107503 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:25.107470    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36a4e29c-hhw62"]
Oct 30 20:51:26.008295 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:26.008252    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:26.284298 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:26.284215    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51a78fcc-db75-4d29-8228-edeb380f36ec" path="/var/lib/kubelet/pods/51a78fcc-db75-4d29-8228-edeb380f36ec/volumes"
Oct 30 20:51:27.577762 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:27.577723    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:27.798361 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:27.798319    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:27.979529 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:27.979445    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:28.253681 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:28.253592    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc0e9f64-d2s85"]
Oct 30 20:51:28.299950 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:28.299921    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dc0e9f64-d2s85"]
Oct 30 20:51:28.413149 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:28.413110    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:28.551907 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:28.551876    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8qvb_91fe9e06-0c30-4000-bddb-2ba89cd048cc/console-operator/2.log"
Oct 30 20:51:28.578851 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:28.568157    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8qvb_91fe9e06-0c30-4000-bddb-2ba89cd048cc/console-operator/2.log"
Oct 30 20:51:28.649123 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:28.649095    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q4f4h_80cae79e-b60f-4efe-bb58-e77753a457e5/ovn-acl-logging/0.log"
Oct 30 20:51:28.659003 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:28.658976    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q4f4h_80cae79e-b60f-4efe-bb58-e77753a457e5/ovn-acl-logging/0.log"
Oct 30 20:51:29.387930 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:29.387891    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:29.582049 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:29.582012    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:30.203929 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:30.203895    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da871b53-aae6-4363-95bb-78bca2b3d48f" path="/var/lib/kubelet/pods/da871b53-aae6-4363-95bb-78bca2b3d48f/volumes"
Oct 30 20:51:30.448067 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:30.448035    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:31.543211 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:31.543158    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:32.930907 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:32.930858    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:34.397245 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:34.397206    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:34.927812 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:34.927552    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:35.199940 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:35.199852    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:35.557920 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:35.557882    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9782c31-vxgfp"]
Oct 30 20:51:35.594850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:35.590070    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9782c31-vxgfp"]
Oct 30 20:51:35.666403 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:35.656892    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:36.205865 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:36.205834    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="559aa737-49ae-4c48-aff0-22a31eaf9698" path="/var/lib/kubelet/pods/559aa737-49ae-4c48-aff0-22a31eaf9698/volumes"
Oct 30 20:51:36.314225 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:36.314189    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:36.583365 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:36.579808    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:36.813933 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:36.813898    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:36.987755 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:36.987667    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:37.171460 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:37.171425    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:37.400903 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:37.400868    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"]
Oct 30 20:51:37.402632 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:37.402593    5071 cpu_manager.go:414] "containerMap: removing container" podUID="52c6baf2-74c7-4ee6-baee-7357190687e5" containerName="payload-processing"
Oct 30 20:51:37.402806 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:37.402790    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52c6baf2-74c7-4ee6-baee-7357190687e5" containerName="payload-processing"
Oct 30 20:51:37.403010 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:37.402994    5071 cpu_manager.go:414] "containerMap: removing container" podUID="7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d" containerName="payload-pre-processing"
Oct 30 20:51:37.403113 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:37.403100    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d" containerName="payload-pre-processing"
Oct 30 20:51:37.403488 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:37.403470    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="52c6baf2-74c7-4ee6-baee-7357190687e5" containerName="payload-processing"
Oct 30 20:51:37.403678 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:37.403656    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a3de0a4-ec0f-4f29-b4bc-c5858e95e73d" containerName="payload-pre-processing"
Oct 30 20:51:37.543556 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:37.543465    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:37.676474 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:37.676381    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78ca07a3-74lzx"]
Oct 30 20:51:37.676972 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:37.676520    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:51:37.837795 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:37.837758    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-b86743-openshift-default-dockercfg-f9z4f\""
Oct 30 20:51:37.862062 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:37.859970    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34c7327-17c9-4085-8970-27dad43ee9be-workload-socket\") pod \"e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx\" (UID: \"f34c7327-17c9-4085-8970-27dad43ee9be\") " pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:51:37.862062 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:37.860101    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34c7327-17c9-4085-8970-27dad43ee9be-credential-socket\") pod \"e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx\" (UID: \"f34c7327-17c9-4085-8970-27dad43ee9be\") " pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:51:37.862062 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:37.860251    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f34c7327-17c9-4085-8970-27dad43ee9be-istiod-ca-cert\") pod \"e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx\" (UID: \"f34c7327-17c9-4085-8970-27dad43ee9be\") " pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:51:37.865399 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:37.864525    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f34c7327-17c9-4085-8970-27dad43ee9be-istio-envoy\") pod \"e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx\" (UID: \"f34c7327-17c9-4085-8970-27dad43ee9be\") " pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:51:37.865399 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:37.864578    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f34c7327-17c9-4085-8970-27dad43ee9be-istio-data\") pod \"e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx\" (UID: \"f34c7327-17c9-4085-8970-27dad43ee9be\") " pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:51:37.865399 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:37.864716    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzv92\" (UniqueName: \"kubernetes.io/projected/f34c7327-17c9-4085-8970-27dad43ee9be-kube-api-access-bzv92\") pod \"e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx\" (UID: \"f34c7327-17c9-4085-8970-27dad43ee9be\") " pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:51:37.865399 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:37.864764    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f34c7327-17c9-4085-8970-27dad43ee9be-workload-certs\") pod \"e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx\" (UID: \"f34c7327-17c9-4085-8970-27dad43ee9be\") " pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:51:37.865399 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:37.864881    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f34c7327-17c9-4085-8970-27dad43ee9be-istio-token\") pod \"e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx\" (UID: \"f34c7327-17c9-4085-8970-27dad43ee9be\") " pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:51:37.865399 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:37.864946    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f34c7327-17c9-4085-8970-27dad43ee9be-istio-podinfo\") pod \"e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx\" (UID: \"f34c7327-17c9-4085-8970-27dad43ee9be\") " pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:51:37.887977 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:37.887942    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78ca07a3-74lzx"]
Oct 30 20:51:37.941051 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:37.940965    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"]
Oct 30 20:51:37.980316 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:37.980275    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzv92\" (UniqueName: \"kubernetes.io/projected/f34c7327-17c9-4085-8970-27dad43ee9be-kube-api-access-bzv92\") pod \"e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx\" (UID: \"f34c7327-17c9-4085-8970-27dad43ee9be\") " pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:51:37.989982 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:37.989941    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f34c7327-17c9-4085-8970-27dad43ee9be-workload-certs\") pod \"e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx\" (UID: \"f34c7327-17c9-4085-8970-27dad43ee9be\") " pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:51:37.990346 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:37.990314    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f34c7327-17c9-4085-8970-27dad43ee9be-istio-token\") pod \"e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx\" (UID: \"f34c7327-17c9-4085-8970-27dad43ee9be\") " pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:51:37.990603 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:37.990581    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f34c7327-17c9-4085-8970-27dad43ee9be-istio-podinfo\") pod \"e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx\" (UID: \"f34c7327-17c9-4085-8970-27dad43ee9be\") " pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:51:37.990861 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:37.990842    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34c7327-17c9-4085-8970-27dad43ee9be-workload-socket\") pod \"e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx\" (UID: \"f34c7327-17c9-4085-8970-27dad43ee9be\") " pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:51:37.991090 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:37.991073    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34c7327-17c9-4085-8970-27dad43ee9be-credential-socket\") pod \"e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx\" (UID: \"f34c7327-17c9-4085-8970-27dad43ee9be\") " pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:51:37.991331 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:37.991317    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f34c7327-17c9-4085-8970-27dad43ee9be-istiod-ca-cert\") pod \"e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx\" (UID: \"f34c7327-17c9-4085-8970-27dad43ee9be\") " pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:51:37.991481 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:37.991466    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f34c7327-17c9-4085-8970-27dad43ee9be-istio-envoy\") pod \"e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx\" (UID: \"f34c7327-17c9-4085-8970-27dad43ee9be\") " pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:51:37.998929 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:37.998898    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f34c7327-17c9-4085-8970-27dad43ee9be-istio-data\") pod \"e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx\" (UID: \"f34c7327-17c9-4085-8970-27dad43ee9be\") " pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:51:37.999242 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:37.993789    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34c7327-17c9-4085-8970-27dad43ee9be-workload-socket\") pod \"e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx\" (UID: \"f34c7327-17c9-4085-8970-27dad43ee9be\") " pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:51:37.999362 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:37.994007    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f34c7327-17c9-4085-8970-27dad43ee9be-workload-certs\") pod \"e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx\" (UID: \"f34c7327-17c9-4085-8970-27dad43ee9be\") " pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:51:37.999662 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:37.995028    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f34c7327-17c9-4085-8970-27dad43ee9be-istiod-ca-cert\") pod \"e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx\" (UID: \"f34c7327-17c9-4085-8970-27dad43ee9be\") " pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:51:38.021895 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:38.000467    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f34c7327-17c9-4085-8970-27dad43ee9be-istio-data\") pod \"e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx\" (UID: \"f34c7327-17c9-4085-8970-27dad43ee9be\") " pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:51:38.022230 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:37.993561    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34c7327-17c9-4085-8970-27dad43ee9be-credential-socket\") pod \"e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx\" (UID: \"f34c7327-17c9-4085-8970-27dad43ee9be\") " pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:51:38.031620 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:38.031551    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f34c7327-17c9-4085-8970-27dad43ee9be-istio-envoy\") pod \"e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx\" (UID: \"f34c7327-17c9-4085-8970-27dad43ee9be\") " pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:51:38.062857 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:38.059598    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f34c7327-17c9-4085-8970-27dad43ee9be-istio-podinfo\") pod \"e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx\" (UID: \"f34c7327-17c9-4085-8970-27dad43ee9be\") " pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:51:38.084865 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:38.078358    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzv92\" (UniqueName: \"kubernetes.io/projected/f34c7327-17c9-4085-8970-27dad43ee9be-kube-api-access-bzv92\") pod \"e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx\" (UID: \"f34c7327-17c9-4085-8970-27dad43ee9be\") " pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:51:38.098057 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:38.098000    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f34c7327-17c9-4085-8970-27dad43ee9be-istio-token\") pod \"e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx\" (UID: \"f34c7327-17c9-4085-8970-27dad43ee9be\") " pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:51:38.170525 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:38.170493    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:51:38.205486 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:38.205397    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43ab1657-d608-474f-b6ed-23ccb44198a7" path="/var/lib/kubelet/pods/43ab1657-d608-474f-b6ed-23ccb44198a7/volumes"
Oct 30 20:51:38.363388 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:38.363351    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:38.601882 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:38.601850    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:38.665076 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:51:38.665047    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf34c7327_17c9_4085_8970_27dad43ee9be.slice/crio-16d7f48ebd3d1fe2b2dcb0fe8a7f1fe67eba0efe495b83cf21dec5e2fdf70eb7 WatchSource:0}: Error finding container 16d7f48ebd3d1fe2b2dcb0fe8a7f1fe67eba0efe495b83cf21dec5e2fdf70eb7: Status 404 returned error can't find the container with id 16d7f48ebd3d1fe2b2dcb0fe8a7f1fe67eba0efe495b83cf21dec5e2fdf70eb7
Oct 30 20:51:38.667926 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:38.667889    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:51:38.668023 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:38.667961    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:51:38.668023 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:38.667997    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:51:38.772396 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:38.772362    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"]
Oct 30 20:51:38.821457 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:38.820962    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:38.830866 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:38.830802    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"]
Oct 30 20:51:38.831106 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:38.831072    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c" podUID="04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d" containerName="istio-proxy" containerID="cri-o://ccfcd46ef63493dcb50fc6a6a0db30229b5804351dc462cd53c2019b2bb3d3c5" gracePeriod=30
Oct 30 20:51:39.053872 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:39.050957    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:39.346553 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:39.346512    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd92310f-n5pd4"]
Oct 30 20:51:39.387481 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:39.382118    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx" event={"ID":"f34c7327-17c9-4085-8970-27dad43ee9be","Type":"ContainerStarted","Data":"73eedcc882576a85936c0f80f504e05da537151d645a1ee4893772a31386424b"}
Oct 30 20:51:39.387481 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:39.382267    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx" event={"ID":"f34c7327-17c9-4085-8970-27dad43ee9be","Type":"ContainerStarted","Data":"16d7f48ebd3d1fe2b2dcb0fe8a7f1fe67eba0efe495b83cf21dec5e2fdf70eb7"}
Oct 30 20:51:39.582402 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:39.571612    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:39.589033 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:39.583141    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd92310f-n5pd4"]
Oct 30 20:51:39.662760 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:39.662110    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"]
Oct 30 20:51:39.685747 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:39.685696    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7" podUID="e5944af0-5719-461a-b5bc-0c05023e086d" containerName="istio-proxy" containerID="cri-o://18cf42390ee7fb76940ea4c26bf7479fe97b6ccda9683fb11884cf0a6f1cbaf6" gracePeriod=30
Oct 30 20:51:39.909896 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:39.906082    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx" podStartSLOduration=2.906062163 podStartE2EDuration="2.906062163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:39.739197743 +0000 UTC m=+1511.656219280" watchObservedRunningTime="2026-10-30 20:51:39.906062163 +0000 UTC m=+1511.823083700"
Oct 30 20:51:39.927658 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:39.925420    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:40.124910 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.124873    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb0460da-vdhn6"]
Oct 30 20:51:40.170849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.167077    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb0460da-vdhn6"]
Oct 30 20:51:40.195275 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.188429    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:51:40.195275 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.188557    5071 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Oct 30 20:51:40.195275 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.188615    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx" podUID="f34c7327-17c9-4085-8970-27dad43ee9be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Oct 30 20:51:40.294862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.288782    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b15da6e-2be7-4689-ae91-1f2b0910f69d" path="/var/lib/kubelet/pods/0b15da6e-2be7-4689-ae91-1f2b0910f69d/volumes"
Oct 30 20:51:40.294862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.289488    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="864ae5b2-2748-4a9e-9ab9-ec3adca7f91a" path="/var/lib/kubelet/pods/864ae5b2-2748-4a9e-9ab9-ec3adca7f91a/volumes"
Oct 30 20:51:40.353681 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.353641    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:40.381850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.356154    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"]
Oct 30 20:51:40.409844 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.405171    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:51:40.576977 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.570696    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ad0ff4fc-openshift-default-dockercfg-hh42l\""
Oct 30 20:51:40.604293 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.604253    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"]
Oct 30 20:51:40.608383 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.608339    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:40.802958 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.801950    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-credential-socket\") pod \"e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw\" (UID: \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\") " pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:51:40.802958 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.802059    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-istio-podinfo\") pod \"e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw\" (UID: \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\") " pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:51:40.802958 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.802130    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9xtm\" (UniqueName: \"kubernetes.io/projected/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-kube-api-access-q9xtm\") pod \"e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw\" (UID: \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\") " pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:51:40.802958 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.802177    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-istiod-ca-cert\") pod \"e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw\" (UID: \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\") " pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:51:40.802958 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.802246    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-workload-certs\") pod \"e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw\" (UID: \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\") " pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:51:40.802958 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.802312    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-istio-data\") pod \"e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw\" (UID: \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\") " pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:51:40.802958 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.802430    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-istio-envoy\") pod \"e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw\" (UID: \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\") " pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:51:40.802958 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.802490    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-istio-token\") pod \"e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw\" (UID: \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\") " pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:51:40.802958 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.802534    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-workload-socket\") pod \"e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw\" (UID: \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\") " pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:51:40.904249 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.904164    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-istio-podinfo\") pod \"e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw\" (UID: \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\") " pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:51:40.904428 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.904273    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9xtm\" (UniqueName: \"kubernetes.io/projected/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-kube-api-access-q9xtm\") pod \"e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw\" (UID: \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\") " pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:51:40.904428 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.904322    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-istiod-ca-cert\") pod \"e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw\" (UID: \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\") " pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:51:40.904428 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.904391    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-workload-certs\") pod \"e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw\" (UID: \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\") " pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:51:40.904609 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.904458    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-istio-data\") pod \"e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw\" (UID: \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\") " pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:51:40.904609 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.904566    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-istio-envoy\") pod \"e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw\" (UID: \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\") " pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:51:40.904751 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.904625    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-istio-token\") pod \"e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw\" (UID: \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\") " pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:51:40.904751 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.904668    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-workload-socket\") pod \"e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw\" (UID: \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\") " pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:51:40.904894 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.904782    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-credential-socket\") pod \"e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw\" (UID: \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\") " pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:51:40.905172 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.905115    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-workload-certs\") pod \"e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw\" (UID: \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\") " pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:51:40.905267 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.905190    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-credential-socket\") pod \"e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw\" (UID: \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\") " pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:51:40.905267 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.905206    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-istiod-ca-cert\") pod \"e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw\" (UID: \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\") " pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:51:40.905483 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.905465    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-istio-data\") pod \"e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw\" (UID: \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\") " pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:51:40.905554 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.905472    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-workload-socket\") pod \"e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw\" (UID: \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\") " pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:51:40.907436 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.907412    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-istio-envoy\") pod \"e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw\" (UID: \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\") " pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:51:40.907978 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.907962    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-istio-podinfo\") pod \"e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw\" (UID: \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\") " pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:51:40.965595 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.965558    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-istio-token\") pod \"e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw\" (UID: \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\") " pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:51:40.968265 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:40.968236    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9xtm\" (UniqueName: \"kubernetes.io/projected/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-kube-api-access-q9xtm\") pod \"e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw\" (UID: \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\") " pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:51:41.070354 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:41.070310    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b54be71f-g6w4m"]
Oct 30 20:51:41.078035 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:41.078005    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b54be71f-g6w4m"]
Oct 30 20:51:41.174691 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:41.172437    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:41.210214 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:41.209035    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:51:41.220414 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:41.220149    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:51:41.360926 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:41.360277    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:41.519222 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:41.519136    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:51:41.520019 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:41.519997    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:51:41.688907 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:41.688872    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"]
Oct 30 20:51:41.702679 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:51:41.702639    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2be68cc_8cd9_4e6e_893e_5d79ef5edea7.slice/crio-07caa6c18f91cd8f5fa72eca89be1cc302cdd9f546bb329432f3426735b5b5fa WatchSource:0}: Error finding container 07caa6c18f91cd8f5fa72eca89be1cc302cdd9f546bb329432f3426735b5b5fa: Status 404 returned error can't find the container with id 07caa6c18f91cd8f5fa72eca89be1cc302cdd9f546bb329432f3426735b5b5fa
Oct 30 20:51:41.705484 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:41.705450    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:51:41.705585 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:41.705519    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:51:41.705585 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:41.705550    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:51:42.218071 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:42.217998    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0951579d-a270-42f5-839a-7a555f465083" path="/var/lib/kubelet/pods/0951579d-a270-42f5-839a-7a555f465083/volumes"
Oct 30 20:51:42.530330 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:42.530228    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw" event={"ID":"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7","Type":"ContainerStarted","Data":"7c0db49e869689b12951479f4020a9c030ee6c81417955579decec1e993e1c60"}
Oct 30 20:51:42.530330 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:42.530296    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw" event={"ID":"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7","Type":"ContainerStarted","Data":"07caa6c18f91cd8f5fa72eca89be1cc302cdd9f546bb329432f3426735b5b5fa"}
Oct 30 20:51:42.658348 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:42.658280    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw" podStartSLOduration=2.658260415 podStartE2EDuration="2.658260415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:42.650645197 +0000 UTC m=+1514.567666735" watchObservedRunningTime="2026-10-30 20:51:42.658260415 +0000 UTC m=+1514.575281948"
Oct 30 20:51:42.896652 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:42.896602    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:43.105772 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:43.105736    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:43.221534 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:43.221455    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:51:43.226927 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:43.226901    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:51:43.540449 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:43.540407    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:51:43.541423 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:43.541402    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:51:44.174495 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.174469    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:51:44.318086 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.318056    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-workload-certs\") pod \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\" (UID: \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\") "
Oct 30 20:51:44.318658 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.318636    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-kube-api-access-7r6sx\" (UniqueName: \"kubernetes.io/projected/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-kube-api-access-7r6sx\") pod \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\" (UID: \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\") "
Oct 30 20:51:44.318788 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.318773    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-istio-data\") pod \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\" (UID: \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\") "
Oct 30 20:51:44.318889 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.318859    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-istio-envoy\") pod \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\" (UID: \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\") "
Oct 30 20:51:44.318957 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.318913    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-istiod-ca-cert\") pod \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\" (UID: \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\") "
Oct 30 20:51:44.319018 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.318956    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-credential-socket\") pod \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\" (UID: \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\") "
Oct 30 20:51:44.319101 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.319083    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-istio-token\" (UniqueName: \"kubernetes.io/projected/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-istio-token\") pod \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\" (UID: \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\") "
Oct 30 20:51:44.319218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.319204    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-istio-podinfo\") pod \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\" (UID: \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\") "
Oct 30 20:51:44.319280 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.319243    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-workload-socket\") pod \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\" (UID: \"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d\") "
Oct 30 20:51:44.320323 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.320288    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-workload-socket" pod "04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d" (UID: "04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:51:44.320575 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.320534    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-workload-certs" pod "04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d" (UID: "04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:51:44.321109 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.321030    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-istio-data" pod "04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d" (UID: "04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:51:44.321226 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.321203    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-istiod-ca-cert" pod "04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d" (UID: "04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:51:44.322035 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.322004    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-credential-socket" pod "04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d" (UID: "04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:51:44.350888 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.323727    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-kube-api-access-7r6sx" pod "04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d" (UID: "04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d"). InnerVolumeSpecName "kube-api-access-7r6sx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:51:44.350888 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.324007    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-istio-envoy" pod "04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d" (UID: "04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:51:44.350888 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.324094    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-istio-token" pod "04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d" (UID: "04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:51:44.350888 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.345480    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-istio-podinfo" pod "04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d" (UID: "04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:51:44.421710 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.421626    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:51:44.421710 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.421654    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:51:44.421710 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.421665    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:51:44.421710 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.421673    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:51:44.421710 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.421682    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:51:44.421710 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.421690    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:51:44.421710 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.421699    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:51:44.421710 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.421707    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r6sx\" (UniqueName: \"kubernetes.io/projected/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-kube-api-access-7r6sx\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:51:44.421710 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.421716    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:51:44.553116 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.553076    5071 generic.go:356] "Generic (PLEG): container finished" podID="04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d" containerID="ccfcd46ef63493dcb50fc6a6a0db30229b5804351dc462cd53c2019b2bb3d3c5" exitCode=0
Oct 30 20:51:44.553315 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.553140    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"
Oct 30 20:51:44.553315 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.553206    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c" event={"ID":"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d","Type":"ContainerDied","Data":"ccfcd46ef63493dcb50fc6a6a0db30229b5804351dc462cd53c2019b2bb3d3c5"}
Oct 30 20:51:44.553315 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.553275    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c" event={"ID":"04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d","Type":"ContainerDied","Data":"acd04669e7a3a30ce8857471c4e0aaa1635dd485de68840bdbed0b6a859198db"}
Oct 30 20:51:44.553315 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.553307    5071 scope.go:122] "RemoveContainer" containerID="ccfcd46ef63493dcb50fc6a6a0db30229b5804351dc462cd53c2019b2bb3d3c5"
Oct 30 20:51:44.567077 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.567057    5071 scope.go:122] "RemoveContainer" containerID="1324533b9d510d2354a3ddc1c62a37116bdc450c04ea33254a87e8877d57ea40"
Oct 30 20:51:44.577770 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.577753    5071 scope.go:122] "RemoveContainer" containerID="ccfcd46ef63493dcb50fc6a6a0db30229b5804351dc462cd53c2019b2bb3d3c5"
Oct 30 20:51:44.578097 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:51:44.578078    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccfcd46ef63493dcb50fc6a6a0db30229b5804351dc462cd53c2019b2bb3d3c5\": container with ID starting with ccfcd46ef63493dcb50fc6a6a0db30229b5804351dc462cd53c2019b2bb3d3c5 not found: ID does not exist" containerID="ccfcd46ef63493dcb50fc6a6a0db30229b5804351dc462cd53c2019b2bb3d3c5"
Oct 30 20:51:44.578155 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.578104    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccfcd46ef63493dcb50fc6a6a0db30229b5804351dc462cd53c2019b2bb3d3c5"} err="failed to get container status \"ccfcd46ef63493dcb50fc6a6a0db30229b5804351dc462cd53c2019b2bb3d3c5\": rpc error: code = NotFound desc = could not find container \"ccfcd46ef63493dcb50fc6a6a0db30229b5804351dc462cd53c2019b2bb3d3c5\": container with ID starting with ccfcd46ef63493dcb50fc6a6a0db30229b5804351dc462cd53c2019b2bb3d3c5 not found: ID does not exist"
Oct 30 20:51:44.578155 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.578121    5071 scope.go:122] "RemoveContainer" containerID="1324533b9d510d2354a3ddc1c62a37116bdc450c04ea33254a87e8877d57ea40"
Oct 30 20:51:44.578399 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:51:44.578380    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1324533b9d510d2354a3ddc1c62a37116bdc450c04ea33254a87e8877d57ea40\": container with ID starting with 1324533b9d510d2354a3ddc1c62a37116bdc450c04ea33254a87e8877d57ea40 not found: ID does not exist" containerID="1324533b9d510d2354a3ddc1c62a37116bdc450c04ea33254a87e8877d57ea40"
Oct 30 20:51:44.578462 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.578410    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1324533b9d510d2354a3ddc1c62a37116bdc450c04ea33254a87e8877d57ea40"} err="failed to get container status \"1324533b9d510d2354a3ddc1c62a37116bdc450c04ea33254a87e8877d57ea40\": rpc error: code = NotFound desc = could not find container \"1324533b9d510d2354a3ddc1c62a37116bdc450c04ea33254a87e8877d57ea40\": container with ID starting with 1324533b9d510d2354a3ddc1c62a37116bdc450c04ea33254a87e8877d57ea40 not found: ID does not exist"
Oct 30 20:51:44.730871 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.730757    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"]
Oct 30 20:51:44.739306 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.739278    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-78ca07a3-openshift-default-74c99c977c-5m72c"]
Oct 30 20:51:44.817996 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:44.817967    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:45.142881 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.142841    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:45.455196 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.455172    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:51:45.551688 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.551659    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5944af0-5719-461a-b5bc-0c05023e086d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5944af0-5719-461a-b5bc-0c05023e086d-istio-envoy\") pod \"e5944af0-5719-461a-b5bc-0c05023e086d\" (UID: \"e5944af0-5719-461a-b5bc-0c05023e086d\") "
Oct 30 20:51:45.551886 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.551727    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5944af0-5719-461a-b5bc-0c05023e086d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5944af0-5719-461a-b5bc-0c05023e086d-istio-data\") pod \"e5944af0-5719-461a-b5bc-0c05023e086d\" (UID: \"e5944af0-5719-461a-b5bc-0c05023e086d\") "
Oct 30 20:51:45.551886 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.551797    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5944af0-5719-461a-b5bc-0c05023e086d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5944af0-5719-461a-b5bc-0c05023e086d-credential-socket\") pod \"e5944af0-5719-461a-b5bc-0c05023e086d\" (UID: \"e5944af0-5719-461a-b5bc-0c05023e086d\") "
Oct 30 20:51:45.551886 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.551834    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5944af0-5719-461a-b5bc-0c05023e086d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5944af0-5719-461a-b5bc-0c05023e086d-workload-certs\") pod \"e5944af0-5719-461a-b5bc-0c05023e086d\" (UID: \"e5944af0-5719-461a-b5bc-0c05023e086d\") "
Oct 30 20:51:45.551886 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.551854    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5944af0-5719-461a-b5bc-0c05023e086d-istio-token\" (UniqueName: \"kubernetes.io/projected/e5944af0-5719-461a-b5bc-0c05023e086d-istio-token\") pod \"e5944af0-5719-461a-b5bc-0c05023e086d\" (UID: \"e5944af0-5719-461a-b5bc-0c05023e086d\") "
Oct 30 20:51:45.552124 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.551965    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5944af0-5719-461a-b5bc-0c05023e086d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5944af0-5719-461a-b5bc-0c05023e086d-istio-podinfo\") pod \"e5944af0-5719-461a-b5bc-0c05023e086d\" (UID: \"e5944af0-5719-461a-b5bc-0c05023e086d\") "
Oct 30 20:51:45.552124 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.551987    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5944af0-5719-461a-b5bc-0c05023e086d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5944af0-5719-461a-b5bc-0c05023e086d-istiod-ca-cert\") pod \"e5944af0-5719-461a-b5bc-0c05023e086d\" (UID: \"e5944af0-5719-461a-b5bc-0c05023e086d\") "
Oct 30 20:51:45.552124 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.552058    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5944af0-5719-461a-b5bc-0c05023e086d-istio-data" pod "e5944af0-5719-461a-b5bc-0c05023e086d" (UID: "e5944af0-5719-461a-b5bc-0c05023e086d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:51:45.552283 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.552180    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5944af0-5719-461a-b5bc-0c05023e086d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5944af0-5719-461a-b5bc-0c05023e086d-workload-socket\") pod \"e5944af0-5719-461a-b5bc-0c05023e086d\" (UID: \"e5944af0-5719-461a-b5bc-0c05023e086d\") "
Oct 30 20:51:45.552283 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.552194    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5944af0-5719-461a-b5bc-0c05023e086d-credential-socket" pod "e5944af0-5719-461a-b5bc-0c05023e086d" (UID: "e5944af0-5719-461a-b5bc-0c05023e086d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:51:45.552283 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.552211    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5944af0-5719-461a-b5bc-0c05023e086d-workload-certs" pod "e5944af0-5719-461a-b5bc-0c05023e086d" (UID: "e5944af0-5719-461a-b5bc-0c05023e086d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:51:45.552283 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.552253    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5944af0-5719-461a-b5bc-0c05023e086d-kube-api-access-5tspx\" (UniqueName: \"kubernetes.io/projected/e5944af0-5719-461a-b5bc-0c05023e086d-kube-api-access-5tspx\") pod \"e5944af0-5719-461a-b5bc-0c05023e086d\" (UID: \"e5944af0-5719-461a-b5bc-0c05023e086d\") "
Oct 30 20:51:45.552518 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.552474    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5944af0-5719-461a-b5bc-0c05023e086d-workload-socket" pod "e5944af0-5719-461a-b5bc-0c05023e086d" (UID: "e5944af0-5719-461a-b5bc-0c05023e086d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:51:45.552518 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.552505    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5944af0-5719-461a-b5bc-0c05023e086d-istiod-ca-cert" pod "e5944af0-5719-461a-b5bc-0c05023e086d" (UID: "e5944af0-5719-461a-b5bc-0c05023e086d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:51:45.553080 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.553010    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5944af0-5719-461a-b5bc-0c05023e086d-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:51:45.553080 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.553038    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5944af0-5719-461a-b5bc-0c05023e086d-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:51:45.553080 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.553059    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5944af0-5719-461a-b5bc-0c05023e086d-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:51:45.553080 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.553071    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5944af0-5719-461a-b5bc-0c05023e086d-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:51:45.553080 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.553080    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5944af0-5719-461a-b5bc-0c05023e086d-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:51:45.554474 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.554448    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5944af0-5719-461a-b5bc-0c05023e086d-istio-token" pod "e5944af0-5719-461a-b5bc-0c05023e086d" (UID: "e5944af0-5719-461a-b5bc-0c05023e086d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:51:45.554474 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.554691    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5944af0-5719-461a-b5bc-0c05023e086d-istio-envoy" pod "e5944af0-5719-461a-b5bc-0c05023e086d" (UID: "e5944af0-5719-461a-b5bc-0c05023e086d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:51:45.554474 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.554762    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5944af0-5719-461a-b5bc-0c05023e086d-istio-podinfo" pod "e5944af0-5719-461a-b5bc-0c05023e086d" (UID: "e5944af0-5719-461a-b5bc-0c05023e086d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:51:45.555082 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.554945    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5944af0-5719-461a-b5bc-0c05023e086d-kube-api-access-5tspx" pod "e5944af0-5719-461a-b5bc-0c05023e086d" (UID: "e5944af0-5719-461a-b5bc-0c05023e086d"). InnerVolumeSpecName "kube-api-access-5tspx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:51:45.567564 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.567539    5071 generic.go:356] "Generic (PLEG): container finished" podID="e5944af0-5719-461a-b5bc-0c05023e086d" containerID="18cf42390ee7fb76940ea4c26bf7479fe97b6ccda9683fb11884cf0a6f1cbaf6" exitCode=0
Oct 30 20:51:45.567649 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.567603    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"
Oct 30 20:51:45.567649 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.567624    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7" event={"ID":"e5944af0-5719-461a-b5bc-0c05023e086d","Type":"ContainerDied","Data":"18cf42390ee7fb76940ea4c26bf7479fe97b6ccda9683fb11884cf0a6f1cbaf6"}
Oct 30 20:51:45.567813 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.567686    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7" event={"ID":"e5944af0-5719-461a-b5bc-0c05023e086d","Type":"ContainerDied","Data":"d1a717db7e7deb0f835d1f82994a9e77a51eb8f8e256fd45f355eb71da3ae3f5"}
Oct 30 20:51:45.567813 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.567709    5071 scope.go:122] "RemoveContainer" containerID="18cf42390ee7fb76940ea4c26bf7479fe97b6ccda9683fb11884cf0a6f1cbaf6"
Oct 30 20:51:45.600421 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.600398    5071 scope.go:122] "RemoveContainer" containerID="33f14c5bd5c7dcf3b1362429cc58fe718aecc5a65d4d44a978c8461a48ed0df5"
Oct 30 20:51:45.612023 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.611999    5071 scope.go:122] "RemoveContainer" containerID="18cf42390ee7fb76940ea4c26bf7479fe97b6ccda9683fb11884cf0a6f1cbaf6"
Oct 30 20:51:45.612341 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:51:45.612314    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18cf42390ee7fb76940ea4c26bf7479fe97b6ccda9683fb11884cf0a6f1cbaf6\": container with ID starting with 18cf42390ee7fb76940ea4c26bf7479fe97b6ccda9683fb11884cf0a6f1cbaf6 not found: ID does not exist" containerID="18cf42390ee7fb76940ea4c26bf7479fe97b6ccda9683fb11884cf0a6f1cbaf6"
Oct 30 20:51:45.612416 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.612358    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18cf42390ee7fb76940ea4c26bf7479fe97b6ccda9683fb11884cf0a6f1cbaf6"} err="failed to get container status \"18cf42390ee7fb76940ea4c26bf7479fe97b6ccda9683fb11884cf0a6f1cbaf6\": rpc error: code = NotFound desc = could not find container \"18cf42390ee7fb76940ea4c26bf7479fe97b6ccda9683fb11884cf0a6f1cbaf6\": container with ID starting with 18cf42390ee7fb76940ea4c26bf7479fe97b6ccda9683fb11884cf0a6f1cbaf6 not found: ID does not exist"
Oct 30 20:51:45.612416 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.612386    5071 scope.go:122] "RemoveContainer" containerID="33f14c5bd5c7dcf3b1362429cc58fe718aecc5a65d4d44a978c8461a48ed0df5"
Oct 30 20:51:45.612696 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:51:45.612672    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33f14c5bd5c7dcf3b1362429cc58fe718aecc5a65d4d44a978c8461a48ed0df5\": container with ID starting with 33f14c5bd5c7dcf3b1362429cc58fe718aecc5a65d4d44a978c8461a48ed0df5 not found: ID does not exist" containerID="33f14c5bd5c7dcf3b1362429cc58fe718aecc5a65d4d44a978c8461a48ed0df5"
Oct 30 20:51:45.612787 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.612767    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33f14c5bd5c7dcf3b1362429cc58fe718aecc5a65d4d44a978c8461a48ed0df5"} err="failed to get container status \"33f14c5bd5c7dcf3b1362429cc58fe718aecc5a65d4d44a978c8461a48ed0df5\": rpc error: code = NotFound desc = could not find container \"33f14c5bd5c7dcf3b1362429cc58fe718aecc5a65d4d44a978c8461a48ed0df5\": container with ID starting with 33f14c5bd5c7dcf3b1362429cc58fe718aecc5a65d4d44a978c8461a48ed0df5 not found: ID does not exist"
Oct 30 20:51:45.654841 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.654798    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5944af0-5719-461a-b5bc-0c05023e086d-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:51:45.655037 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.655017    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tspx\" (UniqueName: \"kubernetes.io/projected/e5944af0-5719-461a-b5bc-0c05023e086d-kube-api-access-5tspx\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:51:45.655163 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.655148    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5944af0-5719-461a-b5bc-0c05023e086d-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:51:45.655283 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.655243    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5944af0-5719-461a-b5bc-0c05023e086d-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:51:45.747369 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.747286    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"]
Oct 30 20:51:45.755237 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.755206    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-bd92310f-openshift-default-6f8cfcc997-lwvb7"]
Oct 30 20:51:45.793803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:45.793763    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:46.039598 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:46.039556    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:46.205175 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:46.205143    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d" path="/var/lib/kubelet/pods/04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d/volumes"
Oct 30 20:51:46.205696 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:46.205680    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5944af0-5719-461a-b5bc-0c05023e086d" path="/var/lib/kubelet/pods/e5944af0-5719-461a-b5bc-0c05023e086d/volumes"
Oct 30 20:51:46.230806 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:46.230773    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:46.488899 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:46.476755    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:46.694226 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:46.694065    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:46.738858 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:46.730629    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b86743-7fbb475c6c-gfq7v"]
Oct 30 20:51:46.738858 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:46.731986    5071 cpu_manager.go:414] "containerMap: removing container" podUID="e5944af0-5719-461a-b5bc-0c05023e086d" containerName="istio-proxy"
Oct 30 20:51:46.738858 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:46.732009    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5944af0-5719-461a-b5bc-0c05023e086d" containerName="istio-proxy"
Oct 30 20:51:46.738858 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:46.732029    5071 cpu_manager.go:414] "containerMap: removing container" podUID="04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d" containerName="istio-proxy"
Oct 30 20:51:46.738858 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:46.732037    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d" containerName="istio-proxy"
Oct 30 20:51:46.738858 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:46.732064    5071 cpu_manager.go:414] "containerMap: removing container" podUID="04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d" containerName="istio-proxy"
Oct 30 20:51:46.738858 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:46.732074    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d" containerName="istio-proxy"
Oct 30 20:51:46.738858 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:46.732128    5071 cpu_manager.go:414] "containerMap: removing container" podUID="e5944af0-5719-461a-b5bc-0c05023e086d" containerName="istio-proxy"
Oct 30 20:51:46.738858 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:46.732136    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5944af0-5719-461a-b5bc-0c05023e086d" containerName="istio-proxy"
Oct 30 20:51:46.776897 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:46.776098    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5944af0-5719-461a-b5bc-0c05023e086d" containerName="istio-proxy"
Oct 30 20:51:46.776897 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:46.776143    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5944af0-5719-461a-b5bc-0c05023e086d" containerName="istio-proxy"
Oct 30 20:51:46.776897 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:46.776160    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d" containerName="istio-proxy"
Oct 30 20:51:46.776897 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:46.776175    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="04a5dcc0-c1bc-4908-bbe0-e36c6dcccb9d" containerName="istio-proxy"
Oct 30 20:51:46.875857 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:46.868572    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b86743-7fbb475c6c-gfq7v"
Oct 30 20:51:46.921892 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:46.876485    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkz4s\" (UniqueName: \"kubernetes.io/projected/9b4b151c-97ec-4a24-9569-53eba1c22d79-kube-api-access-jkz4s\") pod \"payload-pre-processing-e2e-shared-a-w2-b86743-7fbb475c6c-gfq7v\" (UID: \"9b4b151c-97ec-4a24-9569-53eba1c22d79\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b86743-7fbb475c6c-gfq7v"
Oct 30 20:51:46.921892 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:46.876626    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b4b151c-97ec-4a24-9569-53eba1c22d79-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-b86743-7fbb475c6c-gfq7v\" (UID: \"9b4b151c-97ec-4a24-9569-53eba1c22d79\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b86743-7fbb475c6c-gfq7v"
Oct 30 20:51:46.921892 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:46.907804    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b86743-7fbb475c6c-gfq7v"]
Oct 30 20:51:46.980843 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:46.943414    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-b86743-dockercfg-qngxn\""
Oct 30 20:51:47.013878 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:46.985579    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkz4s\" (UniqueName: \"kubernetes.io/projected/9b4b151c-97ec-4a24-9569-53eba1c22d79-kube-api-access-jkz4s\") pod \"payload-pre-processing-e2e-shared-a-w2-b86743-7fbb475c6c-gfq7v\" (UID: \"9b4b151c-97ec-4a24-9569-53eba1c22d79\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b86743-7fbb475c6c-gfq7v"
Oct 30 20:51:47.013878 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:46.985782    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b4b151c-97ec-4a24-9569-53eba1c22d79-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-b86743-7fbb475c6c-gfq7v\" (UID: \"9b4b151c-97ec-4a24-9569-53eba1c22d79\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b86743-7fbb475c6c-gfq7v"
Oct 30 20:51:47.058899 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:51:47.057035    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:51:47.104845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:47.081420    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-b86743\""
Oct 30 20:51:47.104845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:47.081504    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b86743-7c48bdc49b-xdsk6"]
Oct 30 20:51:47.104845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:47.086772    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b4b151c-97ec-4a24-9569-53eba1c22d79-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-b86743-7fbb475c6c-gfq7v\" (UID: \"9b4b151c-97ec-4a24-9569-53eba1c22d79\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b86743-7fbb475c6c-gfq7v"
Oct 30 20:51:47.161353 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:47.139917    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkz4s\" (UniqueName: \"kubernetes.io/projected/9b4b151c-97ec-4a24-9569-53eba1c22d79-kube-api-access-jkz4s\") pod \"payload-pre-processing-e2e-shared-a-w2-b86743-7fbb475c6c-gfq7v\" (UID: \"9b4b151c-97ec-4a24-9569-53eba1c22d79\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b86743-7fbb475c6c-gfq7v"
Oct 30 20:51:47.171845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:47.162922    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b86743-7c48bdc49b-xdsk6"]
Oct 30 20:51:47.175807 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:47.165990    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b86743-7c48bdc49b-xdsk6"
Oct 30 20:51:47.298847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:47.286053    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d3048f3-fbc4-4718-ae16-55b2c6c54890-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b86743-7c48bdc49b-xdsk6\" (UID: \"4d3048f3-fbc4-4718-ae16-55b2c6c54890\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b86743-7c48bdc49b-xdsk6"
Oct 30 20:51:47.298847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:47.286209    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmhxg\" (UniqueName: \"kubernetes.io/projected/4d3048f3-fbc4-4718-ae16-55b2c6c54890-kube-api-access-xmhxg\") pod \"payload-processing-e2e-shared-a-w2-b86743-7c48bdc49b-xdsk6\" (UID: \"4d3048f3-fbc4-4718-ae16-55b2c6c54890\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b86743-7c48bdc49b-xdsk6"
Oct 30 20:51:47.364466 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:47.364127    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b86743-7fbb475c6c-gfq7v"
Oct 30 20:51:47.392578 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:47.392508    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d3048f3-fbc4-4718-ae16-55b2c6c54890-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b86743-7c48bdc49b-xdsk6\" (UID: \"4d3048f3-fbc4-4718-ae16-55b2c6c54890\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b86743-7c48bdc49b-xdsk6"
Oct 30 20:51:47.393813 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:47.393781    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmhxg\" (UniqueName: \"kubernetes.io/projected/4d3048f3-fbc4-4718-ae16-55b2c6c54890-kube-api-access-xmhxg\") pod \"payload-processing-e2e-shared-a-w2-b86743-7c48bdc49b-xdsk6\" (UID: \"4d3048f3-fbc4-4718-ae16-55b2c6c54890\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b86743-7c48bdc49b-xdsk6"
Oct 30 20:51:47.395569 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:47.395521    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d3048f3-fbc4-4718-ae16-55b2c6c54890-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b86743-7c48bdc49b-xdsk6\" (UID: \"4d3048f3-fbc4-4718-ae16-55b2c6c54890\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b86743-7c48bdc49b-xdsk6"
Oct 30 20:51:47.459355 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:47.459323    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmhxg\" (UniqueName: \"kubernetes.io/projected/4d3048f3-fbc4-4718-ae16-55b2c6c54890-kube-api-access-xmhxg\") pod \"payload-processing-e2e-shared-a-w2-b86743-7c48bdc49b-xdsk6\" (UID: \"4d3048f3-fbc4-4718-ae16-55b2c6c54890\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b86743-7c48bdc49b-xdsk6"
Oct 30 20:51:47.575491 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:47.575462    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b86743-7c48bdc49b-xdsk6"
Oct 30 20:51:47.817405 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:51:47.817376    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b4b151c_97ec_4a24_9569_53eba1c22d79.slice/crio-da3bbb0a957d217e3803d4966f7f72aaab060fbaa5f6c018890ece7bb812a73b WatchSource:0}: Error finding container da3bbb0a957d217e3803d4966f7f72aaab060fbaa5f6c018890ece7bb812a73b: Status 404 returned error can't find the container with id da3bbb0a957d217e3803d4966f7f72aaab060fbaa5f6c018890ece7bb812a73b
Oct 30 20:51:47.818909 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:47.818804    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b86743-7fbb475c6c-gfq7v"]
Oct 30 20:51:47.964702 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:51:47.964672    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d3048f3_fbc4_4718_ae16_55b2c6c54890.slice/crio-5aa8799fa8765b98a8453f1d278ee88c04a707738132565cdd8e5caea34173a1 WatchSource:0}: Error finding container 5aa8799fa8765b98a8453f1d278ee88c04a707738132565cdd8e5caea34173a1: Status 404 returned error can't find the container with id 5aa8799fa8765b98a8453f1d278ee88c04a707738132565cdd8e5caea34173a1
Oct 30 20:51:47.965594 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:47.965568    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b86743-7c48bdc49b-xdsk6"]
Oct 30 20:51:48.632326 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:48.632289    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b86743-7c48bdc49b-xdsk6" event={"ID":"4d3048f3-fbc4-4718-ae16-55b2c6c54890","Type":"ContainerStarted","Data":"b1660487fa13d855e4da0aa4d9be71b57ac9154452d88d9a939bb15b5a0712c1"}
Oct 30 20:51:48.632845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:48.632354    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b86743-7c48bdc49b-xdsk6" event={"ID":"4d3048f3-fbc4-4718-ae16-55b2c6c54890","Type":"ContainerStarted","Data":"5aa8799fa8765b98a8453f1d278ee88c04a707738132565cdd8e5caea34173a1"}
Oct 30 20:51:48.632845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:48.632384    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b86743-7c48bdc49b-xdsk6"
Oct 30 20:51:48.645876 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:48.645841    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b86743-7fbb475c6c-gfq7v" event={"ID":"9b4b151c-97ec-4a24-9569-53eba1c22d79","Type":"ContainerStarted","Data":"fd9fddd9a7fba0ded36c2012a5bf760c4a96a03ffa040aa4c05ad5d2d4e5575e"}
Oct 30 20:51:48.646036 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:48.645933    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b86743-7fbb475c6c-gfq7v"
Oct 30 20:51:48.646036 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:48.645957    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b86743-7fbb475c6c-gfq7v" event={"ID":"9b4b151c-97ec-4a24-9569-53eba1c22d79","Type":"ContainerStarted","Data":"da3bbb0a957d217e3803d4966f7f72aaab060fbaa5f6c018890ece7bb812a73b"}
Oct 30 20:51:48.763148 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:48.763088    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b86743-7c48bdc49b-xdsk6" podStartSLOduration=2.763074511 podStartE2EDuration="2.763074511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:48.761116818 +0000 UTC m=+1520.678138336" watchObservedRunningTime="2026-10-30 20:51:48.763074511 +0000 UTC m=+1520.680096046"
Oct 30 20:51:48.888913 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:48.885993    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b86743-7fbb475c6c-gfq7v" podStartSLOduration=2.885973785 podStartE2EDuration="2.885973785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:48.885723767 +0000 UTC m=+1520.802745306" watchObservedRunningTime="2026-10-30 20:51:48.885973785 +0000 UTC m=+1520.802995325"
Oct 30 20:51:50.788844 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:50.787677    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:51.056199 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:51.056115    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44d167d3-9p9hx"]
Oct 30 20:51:51.105519 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:51.105480    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44d167d3-9p9hx"]
Oct 30 20:51:51.435261 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:51.435155    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:51.776475 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:51.776157    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:51.964965 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:51.964934    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ad0ff4fc-768bd4dff9-lcr6k"]
Oct 30 20:51:51.977041 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:51.974961    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad0ff4fc-768bd4dff9-lcr6k"
Oct 30 20:51:52.087904 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:52.033331    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ad0ff4fc\""
Oct 30 20:51:52.087904 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:52.034167    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ad0ff4fc-dockercfg-ndr74\""
Oct 30 20:51:52.087904 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:52.041114    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn52l\" (UniqueName: \"kubernetes.io/projected/07b0cd67-3dfd-47c0-b4d5-df6c3d61946b-kube-api-access-kn52l\") pod \"payload-pre-processing-e2e-ait-ad0ff4fc-768bd4dff9-lcr6k\" (UID: \"07b0cd67-3dfd-47c0-b4d5-df6c3d61946b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ad0ff4fc-768bd4dff9-lcr6k"
Oct 30 20:51:52.133921 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:52.041203    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b0cd67-3dfd-47c0-b4d5-df6c3d61946b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ad0ff4fc-768bd4dff9-lcr6k\" (UID: \"07b0cd67-3dfd-47c0-b4d5-df6c3d61946b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ad0ff4fc-768bd4dff9-lcr6k"
Oct 30 20:51:52.318851 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:52.287421    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn52l\" (UniqueName: \"kubernetes.io/projected/07b0cd67-3dfd-47c0-b4d5-df6c3d61946b-kube-api-access-kn52l\") pod \"payload-pre-processing-e2e-ait-ad0ff4fc-768bd4dff9-lcr6k\" (UID: \"07b0cd67-3dfd-47c0-b4d5-df6c3d61946b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ad0ff4fc-768bd4dff9-lcr6k"
Oct 30 20:51:52.448921 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:52.418008    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b0cd67-3dfd-47c0-b4d5-df6c3d61946b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ad0ff4fc-768bd4dff9-lcr6k\" (UID: \"07b0cd67-3dfd-47c0-b4d5-df6c3d61946b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ad0ff4fc-768bd4dff9-lcr6k"
Oct 30 20:51:52.448921 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:52.419992    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b0cd67-3dfd-47c0-b4d5-df6c3d61946b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ad0ff4fc-768bd4dff9-lcr6k\" (UID: \"07b0cd67-3dfd-47c0-b4d5-df6c3d61946b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ad0ff4fc-768bd4dff9-lcr6k"
Oct 30 20:51:52.462448 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:52.462408    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc73f251-c609-4a35-9757-203c3595c1a8" path="/var/lib/kubelet/pods/cc73f251-c609-4a35-9757-203c3595c1a8/volumes"
Oct 30 20:51:52.472193 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:52.472156    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ad0ff4fc-768bd4dff9-lcr6k"]
Oct 30 20:51:52.479759 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:52.479695    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn52l\" (UniqueName: \"kubernetes.io/projected/07b0cd67-3dfd-47c0-b4d5-df6c3d61946b-kube-api-access-kn52l\") pod \"payload-pre-processing-e2e-ait-ad0ff4fc-768bd4dff9-lcr6k\" (UID: \"07b0cd67-3dfd-47c0-b4d5-df6c3d61946b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ad0ff4fc-768bd4dff9-lcr6k"
Oct 30 20:51:52.490417 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:52.490382    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad0ff4fc-768bd4dff9-lcr6k"
Oct 30 20:51:52.507614 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:52.507561    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ad0ff4fc-98f7856c8-mbzld"]
Oct 30 20:51:52.530629 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:52.529308    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ad0ff4fc-98f7856c8-mbzld"
Oct 30 20:51:52.583087 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:52.580355    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ad0ff4fc-98f7856c8-mbzld"]
Oct 30 20:51:52.589702 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:52.589672    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfd785c6-h5zw5"]
Oct 30 20:51:52.600220 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:52.598160    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfd785c6-h5zw5"]
Oct 30 20:51:52.628733 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:52.628699    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfk72\" (UniqueName: \"kubernetes.io/projected/7947ba4c-bbde-45ff-a524-e00af39df6e6-kube-api-access-dfk72\") pod \"payload-processing-e2e-ait-ad0ff4fc-98f7856c8-mbzld\" (UID: \"7947ba4c-bbde-45ff-a524-e00af39df6e6\") " pod="openshift-ingress/payload-processing-e2e-ait-ad0ff4fc-98f7856c8-mbzld"
Oct 30 20:51:52.628929 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:52.628853    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7947ba4c-bbde-45ff-a524-e00af39df6e6-plugins-config-volume\") pod \"payload-processing-e2e-ait-ad0ff4fc-98f7856c8-mbzld\" (UID: \"7947ba4c-bbde-45ff-a524-e00af39df6e6\") " pod="openshift-ingress/payload-processing-e2e-ait-ad0ff4fc-98f7856c8-mbzld"
Oct 30 20:51:52.731363 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:52.730676    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfk72\" (UniqueName: \"kubernetes.io/projected/7947ba4c-bbde-45ff-a524-e00af39df6e6-kube-api-access-dfk72\") pod \"payload-processing-e2e-ait-ad0ff4fc-98f7856c8-mbzld\" (UID: \"7947ba4c-bbde-45ff-a524-e00af39df6e6\") " pod="openshift-ingress/payload-processing-e2e-ait-ad0ff4fc-98f7856c8-mbzld"
Oct 30 20:51:52.731363 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:52.730785    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7947ba4c-bbde-45ff-a524-e00af39df6e6-plugins-config-volume\") pod \"payload-processing-e2e-ait-ad0ff4fc-98f7856c8-mbzld\" (UID: \"7947ba4c-bbde-45ff-a524-e00af39df6e6\") " pod="openshift-ingress/payload-processing-e2e-ait-ad0ff4fc-98f7856c8-mbzld"
Oct 30 20:51:52.732045 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:52.731953    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7947ba4c-bbde-45ff-a524-e00af39df6e6-plugins-config-volume\") pod \"payload-processing-e2e-ait-ad0ff4fc-98f7856c8-mbzld\" (UID: \"7947ba4c-bbde-45ff-a524-e00af39df6e6\") " pod="openshift-ingress/payload-processing-e2e-ait-ad0ff4fc-98f7856c8-mbzld"
Oct 30 20:51:52.791032 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:52.791002    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfk72\" (UniqueName: \"kubernetes.io/projected/7947ba4c-bbde-45ff-a524-e00af39df6e6-kube-api-access-dfk72\") pod \"payload-processing-e2e-ait-ad0ff4fc-98f7856c8-mbzld\" (UID: \"7947ba4c-bbde-45ff-a524-e00af39df6e6\") " pod="openshift-ingress/payload-processing-e2e-ait-ad0ff4fc-98f7856c8-mbzld"
Oct 30 20:51:52.914486 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:52.914455    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ad0ff4fc-98f7856c8-mbzld"
Oct 30 20:51:52.962174 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:51:52.962137    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b0cd67_3dfd_47c0_b4d5_df6c3d61946b.slice/crio-c441f37741e04e29b37ad6d12d15b9a2758fbc5fea42fddd3222af8f119c59e7 WatchSource:0}: Error finding container c441f37741e04e29b37ad6d12d15b9a2758fbc5fea42fddd3222af8f119c59e7: Status 404 returned error can't find the container with id c441f37741e04e29b37ad6d12d15b9a2758fbc5fea42fddd3222af8f119c59e7
Oct 30 20:51:52.965395 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:52.965369    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ad0ff4fc-768bd4dff9-lcr6k"]
Oct 30 20:51:53.195797 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:53.195752    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:53.308564 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:51:53.308530    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7947ba4c_bbde_45ff_a524_e00af39df6e6.slice/crio-cf4bb4d076ef1d1508faa199f5881102b7449e7de9f0469eb275a5461718954b WatchSource:0}: Error finding container cf4bb4d076ef1d1508faa199f5881102b7449e7de9f0469eb275a5461718954b: Status 404 returned error can't find the container with id cf4bb4d076ef1d1508faa199f5881102b7449e7de9f0469eb275a5461718954b
Oct 30 20:51:53.309954 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:53.309928    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ad0ff4fc-98f7856c8-mbzld"]
Oct 30 20:51:53.809983 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:53.809940    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ad0ff4fc-98f7856c8-mbzld" event={"ID":"7947ba4c-bbde-45ff-a524-e00af39df6e6","Type":"ContainerStarted","Data":"72e7aee37f4e036d1f662d63caf2474f91b8aa4695752aa39d0a027b44e6322a"}
Oct 30 20:51:53.810182 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:53.810012    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ad0ff4fc-98f7856c8-mbzld" event={"ID":"7947ba4c-bbde-45ff-a524-e00af39df6e6","Type":"ContainerStarted","Data":"cf4bb4d076ef1d1508faa199f5881102b7449e7de9f0469eb275a5461718954b"}
Oct 30 20:51:53.810182 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:53.810042    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ad0ff4fc-98f7856c8-mbzld"
Oct 30 20:51:53.811729 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:53.811704    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad0ff4fc-768bd4dff9-lcr6k" event={"ID":"07b0cd67-3dfd-47c0-b4d5-df6c3d61946b","Type":"ContainerStarted","Data":"4d855a5cbaec9919337530b32be1920f1d7c98c913e21e651663a9ae2578611e"}
Oct 30 20:51:53.811874 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:53.811743    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad0ff4fc-768bd4dff9-lcr6k" event={"ID":"07b0cd67-3dfd-47c0-b4d5-df6c3d61946b","Type":"ContainerStarted","Data":"c441f37741e04e29b37ad6d12d15b9a2758fbc5fea42fddd3222af8f119c59e7"}
Oct 30 20:51:53.811874 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:53.811764    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad0ff4fc-768bd4dff9-lcr6k"
Oct 30 20:51:53.929814 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:53.929750    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ad0ff4fc-98f7856c8-mbzld" podStartSLOduration=2.929733041 podStartE2EDuration="2.929733041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:53.925791175 +0000 UTC m=+1525.842812712" watchObservedRunningTime="2026-10-30 20:51:53.929733041 +0000 UTC m=+1525.846754578"
Oct 30 20:51:54.087964 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:54.086928    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad0ff4fc-768bd4dff9-lcr6k" podStartSLOduration=3.086904508 podStartE2EDuration="3.086904508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:54.069352961 +0000 UTC m=+1525.986374498" watchObservedRunningTime="2026-10-30 20:51:54.086904508 +0000 UTC m=+1526.003926045"
Oct 30 20:51:54.229864 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:54.229354    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="073e76de-dacd-49a5-b054-a19c929c0491" path="/var/lib/kubelet/pods/073e76de-dacd-49a5-b054-a19c929c0491/volumes"
Oct 30 20:51:56.066182 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:56.066141    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b447559-rmqt4"]
Oct 30 20:51:56.112522 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:56.112495    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b447559-rmqt4"]
Oct 30 20:51:56.204708 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:56.204674    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5de98b75-5345-48cd-8b72-31d9f5a149b8" path="/var/lib/kubelet/pods/5de98b75-5345-48cd-8b72-31d9f5a149b8/volumes"
Oct 30 20:51:56.577697 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:56.577665    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:51:59.661664 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:59.661632    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b86743-7fbb475c6c-gfq7v"
Oct 30 20:51:59.662164 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:59.661724    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b86743-7c48bdc49b-xdsk6"
Oct 30 20:51:59.997937 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:51:59.997810    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:00.574186 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:00.574148    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"]
Oct 30 20:52:00.582025 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:00.581989    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:52:00.661850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:00.645188    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-0702ff-openshift-default-dockercfg-krdfv\""
Oct 30 20:52:00.661850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:00.648470    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/690bc007-90bc-4e66-8c45-6425aaccc813-workload-certs\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc\" (UID: \"690bc007-90bc-4e66-8c45-6425aaccc813\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:52:00.661850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:00.648507    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/690bc007-90bc-4e66-8c45-6425aaccc813-istiod-ca-cert\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc\" (UID: \"690bc007-90bc-4e66-8c45-6425aaccc813\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:52:00.661850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:00.648762    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27hwp\" (UniqueName: \"kubernetes.io/projected/690bc007-90bc-4e66-8c45-6425aaccc813-kube-api-access-27hwp\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc\" (UID: \"690bc007-90bc-4e66-8c45-6425aaccc813\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:52:00.661850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:00.648876    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/690bc007-90bc-4e66-8c45-6425aaccc813-istio-envoy\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc\" (UID: \"690bc007-90bc-4e66-8c45-6425aaccc813\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:52:00.661850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:00.648952    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/690bc007-90bc-4e66-8c45-6425aaccc813-workload-socket\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc\" (UID: \"690bc007-90bc-4e66-8c45-6425aaccc813\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:52:00.661850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:00.648978    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/690bc007-90bc-4e66-8c45-6425aaccc813-credential-socket\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc\" (UID: \"690bc007-90bc-4e66-8c45-6425aaccc813\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:52:00.661850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:00.649041    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/690bc007-90bc-4e66-8c45-6425aaccc813-istio-token\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc\" (UID: \"690bc007-90bc-4e66-8c45-6425aaccc813\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:52:00.661850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:00.649079    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/690bc007-90bc-4e66-8c45-6425aaccc813-istio-data\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc\" (UID: \"690bc007-90bc-4e66-8c45-6425aaccc813\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:52:00.661850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:00.649110    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/690bc007-90bc-4e66-8c45-6425aaccc813-istio-podinfo\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc\" (UID: \"690bc007-90bc-4e66-8c45-6425aaccc813\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:52:00.761847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:00.760970    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27hwp\" (UniqueName: \"kubernetes.io/projected/690bc007-90bc-4e66-8c45-6425aaccc813-kube-api-access-27hwp\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc\" (UID: \"690bc007-90bc-4e66-8c45-6425aaccc813\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:52:00.761847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:00.761076    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/690bc007-90bc-4e66-8c45-6425aaccc813-istio-envoy\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc\" (UID: \"690bc007-90bc-4e66-8c45-6425aaccc813\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:52:00.761847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:00.761133    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/690bc007-90bc-4e66-8c45-6425aaccc813-workload-socket\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc\" (UID: \"690bc007-90bc-4e66-8c45-6425aaccc813\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:52:00.761847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:00.761155    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/690bc007-90bc-4e66-8c45-6425aaccc813-credential-socket\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc\" (UID: \"690bc007-90bc-4e66-8c45-6425aaccc813\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:52:00.761847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:00.761204    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/690bc007-90bc-4e66-8c45-6425aaccc813-istio-token\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc\" (UID: \"690bc007-90bc-4e66-8c45-6425aaccc813\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:52:00.761847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:00.761236    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/690bc007-90bc-4e66-8c45-6425aaccc813-istio-data\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc\" (UID: \"690bc007-90bc-4e66-8c45-6425aaccc813\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:52:00.761847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:00.761267    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/690bc007-90bc-4e66-8c45-6425aaccc813-istio-podinfo\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc\" (UID: \"690bc007-90bc-4e66-8c45-6425aaccc813\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:52:00.761847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:00.761457    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/690bc007-90bc-4e66-8c45-6425aaccc813-workload-certs\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc\" (UID: \"690bc007-90bc-4e66-8c45-6425aaccc813\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:52:00.761847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:00.761481    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/690bc007-90bc-4e66-8c45-6425aaccc813-istiod-ca-cert\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc\" (UID: \"690bc007-90bc-4e66-8c45-6425aaccc813\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:52:00.798093 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:00.770537    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/690bc007-90bc-4e66-8c45-6425aaccc813-istiod-ca-cert\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc\" (UID: \"690bc007-90bc-4e66-8c45-6425aaccc813\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:52:00.822872 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:00.808389    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:00.899892 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:00.884630    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/690bc007-90bc-4e66-8c45-6425aaccc813-istio-data\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc\" (UID: \"690bc007-90bc-4e66-8c45-6425aaccc813\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:52:00.899892 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:00.884916    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/690bc007-90bc-4e66-8c45-6425aaccc813-workload-socket\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc\" (UID: \"690bc007-90bc-4e66-8c45-6425aaccc813\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:52:00.899892 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:00.885144    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/690bc007-90bc-4e66-8c45-6425aaccc813-credential-socket\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc\" (UID: \"690bc007-90bc-4e66-8c45-6425aaccc813\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:52:00.899892 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:00.885688    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/690bc007-90bc-4e66-8c45-6425aaccc813-workload-certs\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc\" (UID: \"690bc007-90bc-4e66-8c45-6425aaccc813\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:52:01.118776 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:01.026722    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27hwp\" (UniqueName: \"kubernetes.io/projected/690bc007-90bc-4e66-8c45-6425aaccc813-kube-api-access-27hwp\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc\" (UID: \"690bc007-90bc-4e66-8c45-6425aaccc813\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:52:01.256264 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:01.255496    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"]
Oct 30 20:52:01.312697 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:01.298776    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:01.326692 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:01.326615    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/690bc007-90bc-4e66-8c45-6425aaccc813-istio-envoy\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc\" (UID: \"690bc007-90bc-4e66-8c45-6425aaccc813\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:52:01.353115 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:01.351787    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/690bc007-90bc-4e66-8c45-6425aaccc813-istio-podinfo\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc\" (UID: \"690bc007-90bc-4e66-8c45-6425aaccc813\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:52:01.374134 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:01.374103    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/690bc007-90bc-4e66-8c45-6425aaccc813-istio-token\") pod \"e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc\" (UID: \"690bc007-90bc-4e66-8c45-6425aaccc813\") " pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:52:01.647765 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:01.647724    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:52:02.045958 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:02.045918    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:02.107361 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:02.107327    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"]
Oct 30 20:52:02.136014 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:52:02.135965    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod690bc007_90bc_4e66_8c45_6425aaccc813.slice/crio-c4d07e4bf3cb6c3757e33a66491f25c16013a2a9c69063dfc7b30e2bb69a4d0c WatchSource:0}: Error finding container c4d07e4bf3cb6c3757e33a66491f25c16013a2a9c69063dfc7b30e2bb69a4d0c: Status 404 returned error can't find the container with id c4d07e4bf3cb6c3757e33a66491f25c16013a2a9c69063dfc7b30e2bb69a4d0c
Oct 30 20:52:02.158153 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:02.154648    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:52:02.158153 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:02.154802    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:52:02.158153 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:02.154870    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:52:02.270259 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:02.270222    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc" event={"ID":"690bc007-90bc-4e66-8c45-6425aaccc813","Type":"ContainerStarted","Data":"c4d07e4bf3cb6c3757e33a66491f25c16013a2a9c69063dfc7b30e2bb69a4d0c"}
Oct 30 20:52:03.283418 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:03.283382    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc" event={"ID":"690bc007-90bc-4e66-8c45-6425aaccc813","Type":"ContainerStarted","Data":"fbba44203c5c4c37bf690ce7c7fe56f51c0c02777ad9ffb1ecad859af1f262a2"}
Oct 30 20:52:03.507150 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:03.507098    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc" podStartSLOduration=3.5070822550000003 podStartE2EDuration="3.507082255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:03.452329721 +0000 UTC m=+1535.369351257" watchObservedRunningTime="2026-10-30 20:52:03.507082255 +0000 UTC m=+1535.424103791"
Oct 30 20:52:03.650468 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:03.650433    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:52:03.660598 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:03.660065    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:52:04.293393 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:04.293354    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:52:04.296085 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:04.296063    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:52:04.824873 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:04.824843    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ad0ff4fc-98f7856c8-mbzld"
Oct 30 20:52:04.825047 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:04.824916    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad0ff4fc-768bd4dff9-lcr6k"
Oct 30 20:52:06.398589 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:06.398555    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:06.577711 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:06.577675    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:06.752632 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:06.752545    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:07.424057 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:07.424014    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:07.649546 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:07.649511    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:07.742893 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:07.742807    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-822bcb11-6rk2l"]
Oct 30 20:52:07.748104 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:07.748084    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822bcb11-6rk2l"
Oct 30 20:52:07.805551 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:07.805521    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-822bcb11-6rk2l"]
Oct 30 20:52:07.848288 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:07.848246    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 20:52:07.848459 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:07.848291    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:52:07.893702 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:07.893668    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-822bcb11-6rk2l\" (UID: \"acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822bcb11-6rk2l"
Oct 30 20:52:07.893948 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:07.893876    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkwf8\" (UniqueName: \"kubernetes.io/projected/acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5-kube-api-access-mkwf8\") pod \"maas-api-revoke-sub-822bcb11-6rk2l\" (UID: \"acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822bcb11-6rk2l"
Oct 30 20:52:07.894096 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:07.894080    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-822bcb11-6rk2l\" (UID: \"acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822bcb11-6rk2l"
Oct 30 20:52:07.995792 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:07.995713    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-822bcb11-6rk2l\" (UID: \"acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822bcb11-6rk2l"
Oct 30 20:52:07.995965 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:07.995856    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkwf8\" (UniqueName: \"kubernetes.io/projected/acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5-kube-api-access-mkwf8\") pod \"maas-api-revoke-sub-822bcb11-6rk2l\" (UID: \"acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822bcb11-6rk2l"
Oct 30 20:52:07.996048 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:07.995982    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-822bcb11-6rk2l\" (UID: \"acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822bcb11-6rk2l"
Oct 30 20:52:07.996466 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:07.996441    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-822bcb11-6rk2l\" (UID: \"acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822bcb11-6rk2l"
Oct 30 20:52:07.998502 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:07.998484    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-822bcb11-6rk2l\" (UID: \"acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822bcb11-6rk2l"
Oct 30 20:52:08.055565 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:08.055530    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkwf8\" (UniqueName: \"kubernetes.io/projected/acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5-kube-api-access-mkwf8\") pod \"maas-api-revoke-sub-822bcb11-6rk2l\" (UID: \"acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-822bcb11-6rk2l"
Oct 30 20:52:08.063391 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:08.063370    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822bcb11-6rk2l"
Oct 30 20:52:08.451868 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:52:08.451802    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc60fa5_cc15_41bd_b1a0_eee4ebf4a3d5.slice/crio-1ff7e17ceca76c2fc72fcaded316b83d82b2cfa19f844db637142f1381b62c86 WatchSource:0}: Error finding container 1ff7e17ceca76c2fc72fcaded316b83d82b2cfa19f844db637142f1381b62c86: Status 404 returned error can't find the container with id 1ff7e17ceca76c2fc72fcaded316b83d82b2cfa19f844db637142f1381b62c86
Oct 30 20:52:08.454277 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:08.454252    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-822bcb11-6rk2l"]
Oct 30 20:52:09.410680 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:09.410633    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822bcb11-6rk2l" event={"ID":"acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5","Type":"ContainerStarted","Data":"af5244846eaf79c3471770b3684e591473f95020ed88a2cf63f8e9680bc1ba2b"}
Oct 30 20:52:09.410885 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:09.410700    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822bcb11-6rk2l" event={"ID":"acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5","Type":"ContainerStarted","Data":"1ff7e17ceca76c2fc72fcaded316b83d82b2cfa19f844db637142f1381b62c86"}
Oct 30 20:52:09.532988 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:09.532932    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822bcb11-6rk2l" podStartSLOduration=2.532918716 podStartE2EDuration="2.532918716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:09.528095667 +0000 UTC m=+1541.445117214" watchObservedRunningTime="2026-10-30 20:52:09.532918716 +0000 UTC m=+1541.449940251"
Oct 30 20:52:09.641421 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:09.641374    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:09.886868 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:09.878589    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:10.792619 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:10.792580    5071 generic.go:356] "Generic (PLEG): container finished" podID="acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5" containerID="af5244846eaf79c3471770b3684e591473f95020ed88a2cf63f8e9680bc1ba2b" exitCode=0
Oct 30 20:52:10.794093 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:10.793994    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822bcb11-6rk2l" event={"ID":"acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5","Type":"ContainerDied","Data":"af5244846eaf79c3471770b3684e591473f95020ed88a2cf63f8e9680bc1ba2b"}
Oct 30 20:52:12.078921 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:12.077064    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822bcb11-6rk2l"
Oct 30 20:52:12.138619 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:12.138584    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5-maas-api-openshift-ca\") pod \"acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5\" (UID: \"acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5\") "
Oct 30 20:52:12.144908 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:12.144879    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5-kube-api-access-mkwf8\" (UniqueName: \"kubernetes.io/projected/acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5-kube-api-access-mkwf8\") pod \"acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5\" (UID: \"acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5\") "
Oct 30 20:52:12.145536 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:12.144659    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5-maas-api-openshift-ca" pod "acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5" (UID: "acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:52:12.145694 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:12.145678    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5-maas-api-xks-ca\") pod \"acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5\" (UID: \"acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5\") "
Oct 30 20:52:12.146804 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:12.146714    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:12.149369 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:12.149335    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5-maas-api-xks-ca" pod "acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5" (UID: "acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:52:12.151144 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:12.151115    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5-kube-api-access-mkwf8" pod "acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5" (UID: "acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5"). InnerVolumeSpecName "kube-api-access-mkwf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:12.283851 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:12.281544    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkwf8\" (UniqueName: \"kubernetes.io/projected/acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5-kube-api-access-mkwf8\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:12.283851 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:12.281585    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:12.813360 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:12.813334    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822bcb11-6rk2l"
Oct 30 20:52:12.813536 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:12.813361    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-822bcb11-6rk2l" event={"ID":"acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5","Type":"ContainerDied","Data":"1ff7e17ceca76c2fc72fcaded316b83d82b2cfa19f844db637142f1381b62c86"}
Oct 30 20:52:12.813536 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:12.813407    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ff7e17ceca76c2fc72fcaded316b83d82b2cfa19f844db637142f1381b62c86"
Oct 30 20:52:13.207544 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:13.207457    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:15.169163 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:15.169128    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:16.732136 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:52:16.732096    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:52:17.057712 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:17.057674    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25bb2df2-grdd7"]
Oct 30 20:52:17.107788 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:17.107755    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d5a3a4c-8r7gr"]
Oct 30 20:52:17.113226 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:17.113198    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25bb2df2-grdd7"]
Oct 30 20:52:17.117496 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:17.117471    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d5a3a4c-8r7gr"]
Oct 30 20:52:17.966639 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:17.966590    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:18.204388 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:18.204359    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10b290b3-522a-4300-a597-1835714ebda5" path="/var/lib/kubelet/pods/10b290b3-522a-4300-a597-1835714ebda5/volumes"
Oct 30 20:52:18.204898 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:18.204882    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea" path="/var/lib/kubelet/pods/f46b43fd-d482-4a4e-a4fe-dce7f9a7e2ea/volumes"
Oct 30 20:52:18.492451 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:18.492413    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:20.722158 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:20.722124    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:20.944077 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:20.943982    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:22.959684 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:22.959645    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:23.120486 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:23.120403    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af25ff0e-2kjp7"]
Oct 30 20:52:23.130815 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:23.130763    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af25ff0e-2kjp7"]
Oct 30 20:52:24.204101 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:24.204060    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d60d9465-5025-4279-806f-f7a19e55630a" path="/var/lib/kubelet/pods/d60d9465-5025-4279-806f-f7a19e55630a/volumes"
Oct 30 20:52:25.227542 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:25.227504    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:26.106049 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:26.106016    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:52:27.724491 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:27.724458    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm"]
Oct 30 20:52:27.725462 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:27.725436    5071 cpu_manager.go:414] "containerMap: removing container" podUID="acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5" containerName="revoke-keys"
Oct 30 20:52:27.725462 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:27.725461    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5" containerName="revoke-keys"
Oct 30 20:52:27.725628 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:27.725589    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5" containerName="revoke-keys"
Oct 30 20:52:27.730678 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:27.730661    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm"
Oct 30 20:52:27.788685 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:27.788655    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm"]
Oct 30 20:52:27.831450 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:27.831413    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-b86743-serving-cert\""
Oct 30 20:52:27.913899 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:27.912403    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm\" (UID: \"b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm"
Oct 30 20:52:27.913899 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:27.912459    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm\" (UID: \"b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm"
Oct 30 20:52:27.913899 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:27.912489    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5st7b\" (UniqueName: \"kubernetes.io/projected/b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26-kube-api-access-5st7b\") pod \"maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm\" (UID: \"b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm"
Oct 30 20:52:27.913899 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:27.912568    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm\" (UID: \"b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm"
Oct 30 20:52:27.939428 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:27.939400    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:28.040899 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:28.017878    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm\" (UID: \"b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm"
Oct 30 20:52:28.042338 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:28.042310    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm\" (UID: \"b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm"
Oct 30 20:52:28.042546 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:28.042529    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5st7b\" (UniqueName: \"kubernetes.io/projected/b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26-kube-api-access-5st7b\") pod \"maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm\" (UID: \"b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm"
Oct 30 20:52:28.043184 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:28.043161    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm\" (UID: \"b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm"
Oct 30 20:52:28.044279 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:28.044022    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm\" (UID: \"b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm"
Oct 30 20:52:28.072557 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:28.072271    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm\" (UID: \"b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm"
Oct 30 20:52:28.095386 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:28.095354    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm\" (UID: \"b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm"
Oct 30 20:52:28.136125 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:28.128727    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5st7b\" (UniqueName: \"kubernetes.io/projected/b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26-kube-api-access-5st7b\") pod \"maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm\" (UID: \"b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm"
Oct 30 20:52:28.140729 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:28.140695    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:28.351387 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:28.351051    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm"
Oct 30 20:52:28.352548 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:28.352157    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:28.804801 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:52:28.804771    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ca1ec4_c4d4_4b8b_b2d1_3faf33a4ff26.slice/crio-1825750e78ea20a69e56b61fb02aa3875dca03605ffd1fff21d5b3b3c96d9fa0 WatchSource:0}: Error finding container 1825750e78ea20a69e56b61fb02aa3875dca03605ffd1fff21d5b3b3c96d9fa0: Status 404 returned error can't find the container with id 1825750e78ea20a69e56b61fb02aa3875dca03605ffd1fff21d5b3b3c96d9fa0
Oct 30 20:52:28.805258 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:28.804981    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm"]
Oct 30 20:52:28.806567 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:28.806550    5071 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:52:28.991456 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:28.990993    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm" event={"ID":"b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26","Type":"ContainerStarted","Data":"1825750e78ea20a69e56b61fb02aa3875dca03605ffd1fff21d5b3b3c96d9fa0"}
Oct 30 20:52:30.010803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:30.010707    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm" event={"ID":"b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26","Type":"ContainerStarted","Data":"03a0da1ab6c7d26705bd5d8e161c7542afebd31108e09c458b1aae1dae9a08a0"}
Oct 30 20:52:30.012016 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:30.011988    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm"
Oct 30 20:52:30.156428 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:30.156396    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-066357ca-mb8dn"]
Oct 30 20:52:30.249909 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:30.249872    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-066357ca-mb8dn"]
Oct 30 20:52:30.260684 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:30.260608    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm" podStartSLOduration=2.366622012 podStartE2EDuration="3.260587172s" totalImagesPullingTime="893.96516ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:30.191996625 +0000 UTC m=+1562.109018162" watchObservedRunningTime="2026-10-30 20:52:30.260587172 +0000 UTC m=+1562.177608709"
Oct 30 20:52:30.951945 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:30.951912    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:31.856618 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:31.856583    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:32.203857 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:32.203769    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90acbd64-4882-4fce-92cc-7e8b1e4ad884" path="/var/lib/kubelet/pods/90acbd64-4882-4fce-92cc-7e8b1e4ad884/volumes"
Oct 30 20:52:32.558428 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:32.558393    5071 scope.go:122] "RemoveContainer" containerID="0e4e6c7ce87bc203650b6bdfd85df27f6db80ae784995abdcef7e80885f0a37f"
Oct 30 20:52:32.578988 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:32.578965    5071 scope.go:122] "RemoveContainer" containerID="30131cdd25c32bb4cfc36f576f5c4d61b9ea299fa0543c7c6267c8c7cb03d543"
Oct 30 20:52:32.590255 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:32.590235    5071 scope.go:122] "RemoveContainer" containerID="98533b63626ff9bb5f3eddee40cec6509b742f0c9259197b902ee0014c8c1560"
Oct 30 20:52:33.228637 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:33.228598    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:33.390804 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:33.390760    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:35.055181 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:35.055144    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f10c49ba-prq2w"]
Oct 30 20:52:35.104391 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:35.104357    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f10c49ba-prq2w"]
Oct 30 20:52:36.041520 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:36.041494    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm"
Oct 30 20:52:36.212504 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:36.212455    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a0683e4-8c11-4ab5-a025-24f1e93b4d01" path="/var/lib/kubelet/pods/9a0683e4-8c11-4ab5-a025-24f1e93b4d01/volumes"
Oct 30 20:52:36.896808 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:36.896769    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:38.406054 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:38.406013    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:43.376978 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:43.376923    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:44.053804 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:44.053768    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38cb58e4-bpnf4"]
Oct 30 20:52:44.103280 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:44.103250    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38cb58e4-bpnf4"]
Oct 30 20:52:44.203746 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:44.203715    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3218c9e-b073-4137-9c34-26f2d0294ad0" path="/var/lib/kubelet/pods/a3218c9e-b073-4137-9c34-26f2d0294ad0/volumes"
Oct 30 20:52:46.730916 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:52:46.730874    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:52:48.241139 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:48.241094    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:49.145559 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:49.145526    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c08ec9d-5mlgd"]
Oct 30 20:52:49.194573 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:49.194543    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c08ec9d-5mlgd"]
Oct 30 20:52:49.590898 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:49.590857    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:50.306017 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:50.305976    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fde2656-e68e-4dfe-8bde-7138b60889e1" path="/var/lib/kubelet/pods/4fde2656-e68e-4dfe-8bde-7138b60889e1/volumes"
Oct 30 20:52:51.993054 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:51.993016    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk"]
Oct 30 20:52:52.022182 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:52.022151    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk"
Oct 30 20:52:52.361786 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:52.298170    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a367ee40-c5ea-44f3-92c6-d1b5cf1141b3-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pmwpk\" (UID: \"a367ee40-c5ea-44f3-92c6-d1b5cf1141b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk"
Oct 30 20:52:52.361786 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:52.298294    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a367ee40-c5ea-44f3-92c6-d1b5cf1141b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pmwpk\" (UID: \"a367ee40-c5ea-44f3-92c6-d1b5cf1141b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk"
Oct 30 20:52:52.361786 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:52.298332    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a367ee40-c5ea-44f3-92c6-d1b5cf1141b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pmwpk\" (UID: \"a367ee40-c5ea-44f3-92c6-d1b5cf1141b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk"
Oct 30 20:52:52.361786 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:52.298383    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a367ee40-c5ea-44f3-92c6-d1b5cf1141b3-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pmwpk\" (UID: \"a367ee40-c5ea-44f3-92c6-d1b5cf1141b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk"
Oct 30 20:52:52.361786 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:52.298543    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a367ee40-c5ea-44f3-92c6-d1b5cf1141b3-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pmwpk\" (UID: \"a367ee40-c5ea-44f3-92c6-d1b5cf1141b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk"
Oct 30 20:52:52.361786 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:52.298582    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a367ee40-c5ea-44f3-92c6-d1b5cf1141b3-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pmwpk\" (UID: \"a367ee40-c5ea-44f3-92c6-d1b5cf1141b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk"
Oct 30 20:52:52.361786 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:52.298684    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a367ee40-c5ea-44f3-92c6-d1b5cf1141b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pmwpk\" (UID: \"a367ee40-c5ea-44f3-92c6-d1b5cf1141b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk"
Oct 30 20:52:52.361786 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:52.298723    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a367ee40-c5ea-44f3-92c6-d1b5cf1141b3-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pmwpk\" (UID: \"a367ee40-c5ea-44f3-92c6-d1b5cf1141b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk"
Oct 30 20:52:52.361786 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:52.298747    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d4r9\" (UniqueName: \"kubernetes.io/projected/a367ee40-c5ea-44f3-92c6-d1b5cf1141b3-kube-api-access-4d4r9\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pmwpk\" (UID: \"a367ee40-c5ea-44f3-92c6-d1b5cf1141b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk"
Oct 30 20:52:52.450848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:52.424890    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a367ee40-c5ea-44f3-92c6-d1b5cf1141b3-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pmwpk\" (UID: \"a367ee40-c5ea-44f3-92c6-d1b5cf1141b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk"
Oct 30 20:52:52.450848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:52.424955    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a367ee40-c5ea-44f3-92c6-d1b5cf1141b3-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pmwpk\" (UID: \"a367ee40-c5ea-44f3-92c6-d1b5cf1141b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk"
Oct 30 20:52:52.450848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:52.425050    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a367ee40-c5ea-44f3-92c6-d1b5cf1141b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pmwpk\" (UID: \"a367ee40-c5ea-44f3-92c6-d1b5cf1141b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk"
Oct 30 20:52:52.450848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:52.425090    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a367ee40-c5ea-44f3-92c6-d1b5cf1141b3-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pmwpk\" (UID: \"a367ee40-c5ea-44f3-92c6-d1b5cf1141b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk"
Oct 30 20:52:52.450848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:52.425115    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d4r9\" (UniqueName: \"kubernetes.io/projected/a367ee40-c5ea-44f3-92c6-d1b5cf1141b3-kube-api-access-4d4r9\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pmwpk\" (UID: \"a367ee40-c5ea-44f3-92c6-d1b5cf1141b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk"
Oct 30 20:52:52.450848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:52.425217    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a367ee40-c5ea-44f3-92c6-d1b5cf1141b3-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pmwpk\" (UID: \"a367ee40-c5ea-44f3-92c6-d1b5cf1141b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk"
Oct 30 20:52:52.450848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:52.425311    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a367ee40-c5ea-44f3-92c6-d1b5cf1141b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pmwpk\" (UID: \"a367ee40-c5ea-44f3-92c6-d1b5cf1141b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk"
Oct 30 20:52:52.450848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:52.425347    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a367ee40-c5ea-44f3-92c6-d1b5cf1141b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pmwpk\" (UID: \"a367ee40-c5ea-44f3-92c6-d1b5cf1141b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk"
Oct 30 20:52:52.450848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:52.425403    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a367ee40-c5ea-44f3-92c6-d1b5cf1141b3-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pmwpk\" (UID: \"a367ee40-c5ea-44f3-92c6-d1b5cf1141b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk"
Oct 30 20:52:52.450848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:52.440459    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a367ee40-c5ea-44f3-92c6-d1b5cf1141b3-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pmwpk\" (UID: \"a367ee40-c5ea-44f3-92c6-d1b5cf1141b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk"
Oct 30 20:52:52.494353 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:52.461175    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a367ee40-c5ea-44f3-92c6-d1b5cf1141b3-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pmwpk\" (UID: \"a367ee40-c5ea-44f3-92c6-d1b5cf1141b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk"
Oct 30 20:52:52.494353 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:52.461440    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a367ee40-c5ea-44f3-92c6-d1b5cf1141b3-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pmwpk\" (UID: \"a367ee40-c5ea-44f3-92c6-d1b5cf1141b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk"
Oct 30 20:52:52.494353 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:52.477207    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a367ee40-c5ea-44f3-92c6-d1b5cf1141b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pmwpk\" (UID: \"a367ee40-c5ea-44f3-92c6-d1b5cf1141b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk"
Oct 30 20:52:52.496520 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:52.496489    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a367ee40-c5ea-44f3-92c6-d1b5cf1141b3-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pmwpk\" (UID: \"a367ee40-c5ea-44f3-92c6-d1b5cf1141b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk"
Oct 30 20:52:52.684286 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:52.676985    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a367ee40-c5ea-44f3-92c6-d1b5cf1141b3-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pmwpk\" (UID: \"a367ee40-c5ea-44f3-92c6-d1b5cf1141b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk"
Oct 30 20:52:52.712309 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:52.712227    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a367ee40-c5ea-44f3-92c6-d1b5cf1141b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pmwpk\" (UID: \"a367ee40-c5ea-44f3-92c6-d1b5cf1141b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk"
Oct 30 20:52:52.799291 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:52.797309    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:52:52.799291 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:52.798334    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a367ee40-c5ea-44f3-92c6-d1b5cf1141b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pmwpk\" (UID: \"a367ee40-c5ea-44f3-92c6-d1b5cf1141b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk"
Oct 30 20:52:52.799291 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:52.798917    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d4r9\" (UniqueName: \"kubernetes.io/projected/a367ee40-c5ea-44f3-92c6-d1b5cf1141b3-kube-api-access-4d4r9\") pod \"maas-default-gateway-openshift-default-58847f9fc5-pmwpk\" (UID: \"a367ee40-c5ea-44f3-92c6-d1b5cf1141b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk"
Oct 30 20:52:52.814521 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:52.814244    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk"
Oct 30 20:52:52.868807 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:52.854000    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk"]
Oct 30 20:52:53.168040 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:53.166842    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:52:53.451864 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:53.448650    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 30 20:52:53.535868 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:53.521401    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 20:52:53.777679 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:53.777570    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:52:54.050904 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:54.048036    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:52:54.124866 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:54.064916    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:52:54.215927 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:54.185252    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 20:52:54.215927 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:54.185848    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:52:54.218416 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:54.216616    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e8761ca-97rdd"]
Oct 30 20:52:54.224418 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:54.223143    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:54.459324 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:54.332387    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:52:54.459324 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:54.332487    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:52:54.459324 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:54.332530    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:52:54.459530 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:54.459417    5071 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:52:54.496699 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:54.495939    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:52:54.705364 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:54.705316    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:52:54.820618 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:54.739073    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:52:54.820618 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:54.739122    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e8761ca-97rdd"]
Oct 30 20:52:54.820618 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:54.812933    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:52:54.824430 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:54.824025    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk"]
Oct 30 20:52:54.904586 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:54.904420    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 20:52:55.312983 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:55.312284    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk" event={"ID":"a367ee40-c5ea-44f3-92c6-d1b5cf1141b3","Type":"ContainerStarted","Data":"fed9bd4de7d6106b5d4f7a515eab80aafa64b15c04e760c63c59ea1efe84c982"}
Oct 30 20:52:56.074845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:56.071484    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:56.211044 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:56.211010    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8205cd06-0a45-4d84-99cb-0c0ca310a887" path="/var/lib/kubelet/pods/8205cd06-0a45-4d84-99cb-0c0ca310a887/volumes"
Oct 30 20:52:56.321460 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:56.321428    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk" event={"ID":"a367ee40-c5ea-44f3-92c6-d1b5cf1141b3","Type":"ContainerStarted","Data":"ccc32a7cd7bf2c7c3294030219201398133ebf20385d2dd635b84a705d4c0094"}
Oct 30 20:52:56.566848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:56.552398    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk" podStartSLOduration=5.552375463 podStartE2EDuration="5.552375463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:56.494008254 +0000 UTC m=+1588.411029791" watchObservedRunningTime="2026-10-30 20:52:56.552375463 +0000 UTC m=+1588.469397000"
Oct 30 20:52:56.823633 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:56.823496    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk"
Oct 30 20:52:56.824929 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:56.824892    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-pmwpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 20:52:56.825073 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:56.824949    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk" podUID="a367ee40-c5ea-44f3-92c6-d1b5cf1141b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 20:52:57.087139 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:57.087020    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-632ebc82-fggb4"]
Oct 30 20:52:57.095608 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:57.095570    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-632ebc82-fggb4"]
Oct 30 20:52:57.823191 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:57.823162    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-pmwpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 20:52:57.823585 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:57.823215    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk" podUID="a367ee40-c5ea-44f3-92c6-d1b5cf1141b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 20:52:58.207523 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:58.207449    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c164e31-eaeb-49fa-bcae-17934ed9febc" path="/var/lib/kubelet/pods/5c164e31-eaeb-49fa-bcae-17934ed9febc/volumes"
Oct 30 20:52:58.823730 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:58.823688    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-pmwpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 20:52:58.824246 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:58.823756    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk" podUID="a367ee40-c5ea-44f3-92c6-d1b5cf1141b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 20:52:59.426311 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:59.426275    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws"]
Oct 30 20:52:59.449218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:59.449188    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws"
Oct 30 20:52:59.492403 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:59.492372    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws"]
Oct 30 20:52:59.506335 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:59.506294    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bac9ec7-daeb-43cd-afb4-c8aca96f77a2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws\" (UID: \"6bac9ec7-daeb-43cd-afb4-c8aca96f77a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws"
Oct 30 20:52:59.506335 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:59.506407    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bac9ec7-daeb-43cd-afb4-c8aca96f77a2-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws\" (UID: \"6bac9ec7-daeb-43cd-afb4-c8aca96f77a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws"
Oct 30 20:52:59.506714 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:59.506517    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc7w8\" (UniqueName: \"kubernetes.io/projected/6bac9ec7-daeb-43cd-afb4-c8aca96f77a2-kube-api-access-qc7w8\") pod \"maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws\" (UID: \"6bac9ec7-daeb-43cd-afb4-c8aca96f77a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws"
Oct 30 20:52:59.506714 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:59.506565    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bac9ec7-daeb-43cd-afb4-c8aca96f77a2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws\" (UID: \"6bac9ec7-daeb-43cd-afb4-c8aca96f77a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws"
Oct 30 20:52:59.550404 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:59.536047    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-0702ff-serving-cert\""
Oct 30 20:52:59.591019 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:59.590981    5071 patch_prober.go:29] interesting pod/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:52:59.591209 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:59.591059    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw" podUID="f2be68cc-8cd9-4e6e-893e-5d79ef5edea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:52:59.608995 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:59.608366    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bac9ec7-daeb-43cd-afb4-c8aca96f77a2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws\" (UID: \"6bac9ec7-daeb-43cd-afb4-c8aca96f77a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws"
Oct 30 20:52:59.608995 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:59.608492    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bac9ec7-daeb-43cd-afb4-c8aca96f77a2-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws\" (UID: \"6bac9ec7-daeb-43cd-afb4-c8aca96f77a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws"
Oct 30 20:52:59.608995 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:52:59.608604    5071 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0702ff-serving-cert: secret "maas-api-e2e-shared-a-w5-0702ff-serving-cert" not found
Oct 30 20:52:59.608995 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:59.608624    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc7w8\" (UniqueName: \"kubernetes.io/projected/6bac9ec7-daeb-43cd-afb4-c8aca96f77a2-kube-api-access-qc7w8\") pod \"maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws\" (UID: \"6bac9ec7-daeb-43cd-afb4-c8aca96f77a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws"
Oct 30 20:52:59.608995 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:52:59.608677    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bac9ec7-daeb-43cd-afb4-c8aca96f77a2-maas-api-tls podName:6bac9ec7-daeb-43cd-afb4-c8aca96f77a2 nodeName:}" failed. No retries permitted until 2026-10-30 20:53:00.108653729 +0000 UTC m=+1592.025675263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6bac9ec7-daeb-43cd-afb4-c8aca96f77a2-maas-api-tls") pod "maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws" (UID: "6bac9ec7-daeb-43cd-afb4-c8aca96f77a2") : secret "maas-api-e2e-shared-a-w5-0702ff-serving-cert" not found
Oct 30 20:52:59.608995 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:59.608722    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bac9ec7-daeb-43cd-afb4-c8aca96f77a2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws\" (UID: \"6bac9ec7-daeb-43cd-afb4-c8aca96f77a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws"
Oct 30 20:52:59.609944 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:59.609675    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bac9ec7-daeb-43cd-afb4-c8aca96f77a2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws\" (UID: \"6bac9ec7-daeb-43cd-afb4-c8aca96f77a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws"
Oct 30 20:52:59.616608 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:59.616578    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bac9ec7-daeb-43cd-afb4-c8aca96f77a2-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws\" (UID: \"6bac9ec7-daeb-43cd-afb4-c8aca96f77a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws"
Oct 30 20:52:59.681243 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:59.668564    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:52:59.691700 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:59.691614    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc7w8\" (UniqueName: \"kubernetes.io/projected/6bac9ec7-daeb-43cd-afb4-c8aca96f77a2-kube-api-access-qc7w8\") pod \"maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws\" (UID: \"6bac9ec7-daeb-43cd-afb4-c8aca96f77a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws"
Oct 30 20:52:59.834976 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:59.827422    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-pmwpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 20:52:59.834976 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:59.827498    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk" podUID="a367ee40-c5ea-44f3-92c6-d1b5cf1141b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 20:52:59.872846 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:52:59.859420    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:00.121407 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:00.121369    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bac9ec7-daeb-43cd-afb4-c8aca96f77a2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws\" (UID: \"6bac9ec7-daeb-43cd-afb4-c8aca96f77a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws"
Oct 30 20:53:00.129458 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:53:00.129425    5071 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0702ff-serving-cert: secret "maas-api-e2e-shared-a-w5-0702ff-serving-cert" not found
Oct 30 20:53:00.129743 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:53:00.129728    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bac9ec7-daeb-43cd-afb4-c8aca96f77a2-maas-api-tls podName:6bac9ec7-daeb-43cd-afb4-c8aca96f77a2 nodeName:}" failed. No retries permitted until 2026-10-30 20:53:01.129704385 +0000 UTC m=+1593.046725918 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6bac9ec7-daeb-43cd-afb4-c8aca96f77a2-maas-api-tls") pod "maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws" (UID: "6bac9ec7-daeb-43cd-afb4-c8aca96f77a2") : secret "maas-api-e2e-shared-a-w5-0702ff-serving-cert" not found
Oct 30 20:53:00.524921 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:00.524813    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:00.823591 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:00.823559    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-pmwpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 20:53:00.823768 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:00.823614    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk" podUID="a367ee40-c5ea-44f3-92c6-d1b5cf1141b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 20:53:01.179330 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:01.179233    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bac9ec7-daeb-43cd-afb4-c8aca96f77a2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws\" (UID: \"6bac9ec7-daeb-43cd-afb4-c8aca96f77a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws"
Oct 30 20:53:01.182308 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:01.182273    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bac9ec7-daeb-43cd-afb4-c8aca96f77a2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws\" (UID: \"6bac9ec7-daeb-43cd-afb4-c8aca96f77a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws"
Oct 30 20:53:01.281737 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:01.281155    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws"
Oct 30 20:53:01.838132 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:01.838094    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-pmwpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 20:53:01.839173 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:01.839048    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk" podUID="a367ee40-c5ea-44f3-92c6-d1b5cf1141b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 20:53:01.840034 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:01.840016    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws"]
Oct 30 20:53:02.419442 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:02.419398    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws" event={"ID":"6bac9ec7-daeb-43cd-afb4-c8aca96f77a2","Type":"ContainerStarted","Data":"96c78513a64286df4f6d8d77def63b73ca9a4a72e971f5e37d49b85ae59bf822"}
Oct 30 20:53:02.828883 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:02.826645    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-pmwpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 20:53:02.828883 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:02.826709    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk" podUID="a367ee40-c5ea-44f3-92c6-d1b5cf1141b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 20:53:03.433015 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:03.432909    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws" event={"ID":"6bac9ec7-daeb-43cd-afb4-c8aca96f77a2","Type":"ContainerStarted","Data":"b490c6207f4c2258bdb7e8f737e745cd67ff200ec14f77324426032d8b755c1c"}
Oct 30 20:53:03.433015 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:03.433013    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws"
Oct 30 20:53:03.558539 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:03.558465    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws" podStartSLOduration=3.282199192 podStartE2EDuration="4.558444747s" totalImagesPullingTime="1.276245555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:03.550410325 +0000 UTC m=+1595.467431862" watchObservedRunningTime="2026-10-30 20:53:03.558444747 +0000 UTC m=+1595.475466284"
Oct 30 20:53:03.837128 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:03.835252    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-pmwpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Oct 30 20:53:03.837128 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:03.835317    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk" podUID="a367ee40-c5ea-44f3-92c6-d1b5cf1141b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Oct 30 20:53:04.066243 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:04.066204    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3114635a-nvgjq"]
Oct 30 20:53:04.202112 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:04.074579    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3114635a-nvgjq"]
Oct 30 20:53:04.220331 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:04.206903    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb2c533f-9768-436d-bddc-4de7e46885ec" path="/var/lib/kubelet/pods/cb2c533f-9768-436d-bddc-4de7e46885ec/volumes"
Oct 30 20:53:04.449665 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:04.449639    5071 generic.go:356] "Generic (PLEG): container finished" podID="f2be68cc-8cd9-4e6e-893e-5d79ef5edea7" containerID="7c0db49e869689b12951479f4020a9c030ee6c81417955579decec1e993e1c60" exitCode=0
Oct 30 20:53:04.450147 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:04.449716    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw" event={"ID":"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7","Type":"ContainerDied","Data":"7c0db49e869689b12951479f4020a9c030ee6c81417955579decec1e993e1c60"}
Oct 30 20:53:04.450332 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:04.450310    5071 scope.go:122] "RemoveContainer" containerID="7c0db49e869689b12951479f4020a9c030ee6c81417955579decec1e993e1c60"
Oct 30 20:53:04.452299 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:04.452238    5071 generic.go:356] "Generic (PLEG): container finished" podID="f34c7327-17c9-4085-8970-27dad43ee9be" containerID="73eedcc882576a85936c0f80f504e05da537151d645a1ee4893772a31386424b" exitCode=0
Oct 30 20:53:04.452299 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:04.452266    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:53:04.452486 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:04.452308    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx" event={"ID":"f34c7327-17c9-4085-8970-27dad43ee9be","Type":"ContainerDied","Data":"73eedcc882576a85936c0f80f504e05da537151d645a1ee4893772a31386424b"}
Oct 30 20:53:04.452486 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:04.452353    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:53:04.452486 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:04.452403    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:53:04.452719 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:04.452698    5071 scope.go:122] "RemoveContainer" containerID="73eedcc882576a85936c0f80f504e05da537151d645a1ee4893772a31386424b"
Oct 30 20:53:04.454409 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:04.454371    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:53:04.454522 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:04.454454    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:53:04.454522 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:04.454502    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:53:05.066765 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:05.066586    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddcb2e1b-qbrbt"]
Oct 30 20:53:05.075925 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:05.075547    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddcb2e1b-qbrbt"]
Oct 30 20:53:05.339597 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:05.339514    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:05.530847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:05.530393    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx" event={"ID":"f34c7327-17c9-4085-8970-27dad43ee9be","Type":"ContainerStarted","Data":"6e2c482fe042adf988fd644bae9f0fc85ab95e78f063b0536de8b7cc5d9237dc"}
Oct 30 20:53:05.570883 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:05.563435    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw" event={"ID":"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7","Type":"ContainerStarted","Data":"d5490695c2967cfa103ba2b6cb191fc835e8c43ab6e556a3b0b922c1b615b5c8"}
Oct 30 20:53:05.642026 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:05.635177    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk"
Oct 30 20:53:05.642026 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:05.636916    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk"
Oct 30 20:53:05.646421 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:05.646393    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk"
Oct 30 20:53:06.120374 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:06.120337    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51daf491-bb7cc"]
Oct 30 20:53:06.125439 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:06.125411    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51daf491-bb7cc"]
Oct 30 20:53:06.186848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:06.181233    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:53:06.186848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:06.181411    5071 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Oct 30 20:53:06.186848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:06.181456    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx" podUID="f34c7327-17c9-4085-8970-27dad43ee9be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Oct 30 20:53:06.208963 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:06.208934    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64d24113-5574-4a4a-9bf3-0c2814605103" path="/var/lib/kubelet/pods/64d24113-5574-4a4a-9bf3-0c2814605103/volumes"
Oct 30 20:53:06.209414 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:06.209401    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a90f40d0-8846-469d-b3e9-c9158d667dfa" path="/var/lib/kubelet/pods/a90f40d0-8846-469d-b3e9-c9158d667dfa/volumes"
Oct 30 20:53:06.221223 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:06.221198    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:53:06.221395 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:06.221378    5071 patch_prober.go:29] interesting pod/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 20:53:06.221447 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:06.221427    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw" podUID="f2be68cc-8cd9-4e6e-893e-5d79ef5edea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 20:53:07.177845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:07.175197    5071 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Oct 30 20:53:07.177845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:07.175274    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx" podUID="f34c7327-17c9-4085-8970-27dad43ee9be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Oct 30 20:53:07.221053 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:07.221022    5071 patch_prober.go:29] interesting pod/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 20:53:07.221212 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:07.221073    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw" podUID="f2be68cc-8cd9-4e6e-893e-5d79ef5edea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 20:53:08.178846 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:08.178202    5071 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Oct 30 20:53:08.178846 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:08.178272    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx" podUID="f34c7327-17c9-4085-8970-27dad43ee9be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Oct 30 20:53:08.221477 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:08.221448    5071 patch_prober.go:29] interesting pod/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 20:53:08.221656 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:08.221514    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw" podUID="f2be68cc-8cd9-4e6e-893e-5d79ef5edea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 20:53:09.178849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:09.178725    5071 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Oct 30 20:53:09.178849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:09.178786    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx" podUID="f34c7327-17c9-4085-8970-27dad43ee9be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Oct 30 20:53:09.221105 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:09.221064    5071 patch_prober.go:29] interesting pod/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 20:53:09.221322 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:09.221301    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw" podUID="f2be68cc-8cd9-4e6e-893e-5d79ef5edea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 20:53:09.485010 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:09.484923    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws"
Oct 30 20:53:10.173242 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:10.173204    5071 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Oct 30 20:53:10.173563 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:10.173532    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx" podUID="f34c7327-17c9-4085-8970-27dad43ee9be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Oct 30 20:53:10.221428 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:10.221402    5071 patch_prober.go:29] interesting pod/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 20:53:10.221883 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:10.221861    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw" podUID="f2be68cc-8cd9-4e6e-893e-5d79ef5edea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 20:53:10.639053 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:10.639018    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:11.173247 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:11.173209    5071 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Oct 30 20:53:11.173557 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:11.173530    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx" podUID="f34c7327-17c9-4085-8970-27dad43ee9be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Oct 30 20:53:11.221302 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:11.221266    5071 patch_prober.go:29] interesting pod/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 20:53:11.221478 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:11.221318    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw" podUID="f2be68cc-8cd9-4e6e-893e-5d79ef5edea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 20:53:11.519966 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:11.519887    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:53:12.176266 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:12.176227    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:53:12.177199 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:12.177177    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:53:12.225701 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:12.225673    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:53:12.226168 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:12.225958    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:53:12.226740 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:12.226725    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:53:14.054728 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:14.054694    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a4c3808-fzrrm"]
Oct 30 20:53:14.104568 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:14.104536    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f60b94db-5tt7f"]
Oct 30 20:53:14.109193 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:14.109171    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a4c3808-fzrrm"]
Oct 30 20:53:14.113191 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:14.113167    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f60b94db-5tt7f"]
Oct 30 20:53:14.204396 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:14.204362    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ac57fcd-364e-4d13-a6f9-0308d322bc32" path="/var/lib/kubelet/pods/0ac57fcd-364e-4d13-a6f9-0308d322bc32/volumes"
Oct 30 20:53:14.204881 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:14.204866    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf0d87d0-a8e9-41a4-ae83-18479e8aadd8" path="/var/lib/kubelet/pods/cf0d87d0-a8e9-41a4-ae83-18479e8aadd8/volumes"
Oct 30 20:53:15.149502 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:15.149466    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:15.772862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:15.768389    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:15.910847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:15.910348    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"]
Oct 30 20:53:15.992948 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:15.992914    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:53:16.003114 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:16.003086    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"]
Oct 30 20:53:16.061289 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:16.061246    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:16.075261 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:16.065571    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d204447d-7147-415b-abf8-853f68b51fec-istio-podinfo\") pod \"e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt\" (UID: \"d204447d-7147-415b-abf8-853f68b51fec\") " pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:53:16.075261 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:16.065684    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d204447d-7147-415b-abf8-853f68b51fec-istio-token\") pod \"e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt\" (UID: \"d204447d-7147-415b-abf8-853f68b51fec\") " pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:53:16.132729 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:16.131110    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d204447d-7147-415b-abf8-853f68b51fec-workload-socket\") pod \"e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt\" (UID: \"d204447d-7147-415b-abf8-853f68b51fec\") " pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:53:16.132729 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:16.131178    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d204447d-7147-415b-abf8-853f68b51fec-istio-data\") pod \"e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt\" (UID: \"d204447d-7147-415b-abf8-853f68b51fec\") " pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:53:16.132729 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:16.131246    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d204447d-7147-415b-abf8-853f68b51fec-workload-certs\") pod \"e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt\" (UID: \"d204447d-7147-415b-abf8-853f68b51fec\") " pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:53:16.132729 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:16.131334    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lmc2\" (UniqueName: \"kubernetes.io/projected/d204447d-7147-415b-abf8-853f68b51fec-kube-api-access-7lmc2\") pod \"e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt\" (UID: \"d204447d-7147-415b-abf8-853f68b51fec\") " pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:53:16.132729 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:16.131360    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d204447d-7147-415b-abf8-853f68b51fec-istiod-ca-cert\") pod \"e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt\" (UID: \"d204447d-7147-415b-abf8-853f68b51fec\") " pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:53:16.132729 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:16.131588    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d204447d-7147-415b-abf8-853f68b51fec-credential-socket\") pod \"e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt\" (UID: \"d204447d-7147-415b-abf8-853f68b51fec\") " pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:53:16.132729 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:16.131708    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d204447d-7147-415b-abf8-853f68b51fec-istio-envoy\") pod \"e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt\" (UID: \"d204447d-7147-415b-abf8-853f68b51fec\") " pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:53:16.242521 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:16.242496    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-0c83a089-openshift-default-dockercfg-2vxts\""
Oct 30 20:53:16.256721 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:16.256687    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d204447d-7147-415b-abf8-853f68b51fec-credential-socket\") pod \"e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt\" (UID: \"d204447d-7147-415b-abf8-853f68b51fec\") " pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:53:16.260920 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:16.257086    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d204447d-7147-415b-abf8-853f68b51fec-istio-envoy\") pod \"e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt\" (UID: \"d204447d-7147-415b-abf8-853f68b51fec\") " pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:53:16.261086 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:16.260992    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d204447d-7147-415b-abf8-853f68b51fec-istio-podinfo\") pod \"e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt\" (UID: \"d204447d-7147-415b-abf8-853f68b51fec\") " pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:53:16.261153 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:16.261105    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d204447d-7147-415b-abf8-853f68b51fec-istio-token\") pod \"e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt\" (UID: \"d204447d-7147-415b-abf8-853f68b51fec\") " pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:53:16.261479 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:16.261453    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d204447d-7147-415b-abf8-853f68b51fec-workload-socket\") pod \"e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt\" (UID: \"d204447d-7147-415b-abf8-853f68b51fec\") " pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:53:16.261584 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:16.261506    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d204447d-7147-415b-abf8-853f68b51fec-istio-data\") pod \"e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt\" (UID: \"d204447d-7147-415b-abf8-853f68b51fec\") " pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:53:16.261584 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:16.261574    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d204447d-7147-415b-abf8-853f68b51fec-workload-certs\") pod \"e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt\" (UID: \"d204447d-7147-415b-abf8-853f68b51fec\") " pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:53:16.262106 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:16.262076    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lmc2\" (UniqueName: \"kubernetes.io/projected/d204447d-7147-415b-abf8-853f68b51fec-kube-api-access-7lmc2\") pod \"e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt\" (UID: \"d204447d-7147-415b-abf8-853f68b51fec\") " pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:53:16.262213 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:16.262136    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d204447d-7147-415b-abf8-853f68b51fec-istiod-ca-cert\") pod \"e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt\" (UID: \"d204447d-7147-415b-abf8-853f68b51fec\") " pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:53:16.264541 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:16.264513    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d204447d-7147-415b-abf8-853f68b51fec-istiod-ca-cert\") pod \"e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt\" (UID: \"d204447d-7147-415b-abf8-853f68b51fec\") " pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:53:16.273877 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:16.273792    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d204447d-7147-415b-abf8-853f68b51fec-workload-certs\") pod \"e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt\" (UID: \"d204447d-7147-415b-abf8-853f68b51fec\") " pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:53:16.274261 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:16.274236    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d204447d-7147-415b-abf8-853f68b51fec-istio-data\") pod \"e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt\" (UID: \"d204447d-7147-415b-abf8-853f68b51fec\") " pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:53:16.274571 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:16.274549    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d204447d-7147-415b-abf8-853f68b51fec-credential-socket\") pod \"e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt\" (UID: \"d204447d-7147-415b-abf8-853f68b51fec\") " pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:53:16.277667 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:16.277044    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d204447d-7147-415b-abf8-853f68b51fec-istio-podinfo\") pod \"e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt\" (UID: \"d204447d-7147-415b-abf8-853f68b51fec\") " pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:53:16.286202 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:16.286173    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d204447d-7147-415b-abf8-853f68b51fec-workload-socket\") pod \"e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt\" (UID: \"d204447d-7147-415b-abf8-853f68b51fec\") " pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:53:16.288667 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:16.288634    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d204447d-7147-415b-abf8-853f68b51fec-istio-envoy\") pod \"e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt\" (UID: \"d204447d-7147-415b-abf8-853f68b51fec\") " pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:53:16.424897 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:16.424381    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lmc2\" (UniqueName: \"kubernetes.io/projected/d204447d-7147-415b-abf8-853f68b51fec-kube-api-access-7lmc2\") pod \"e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt\" (UID: \"d204447d-7147-415b-abf8-853f68b51fec\") " pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:53:16.513850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:16.512292    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d204447d-7147-415b-abf8-853f68b51fec-istio-token\") pod \"e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt\" (UID: \"d204447d-7147-415b-abf8-853f68b51fec\") " pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:53:16.519876 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:16.516234    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:53:16.611940 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:16.610626    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:16.737965 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:53:16.737926    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:53:17.075331 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:53:17.075296    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd204447d_7147_415b_abf8_853f68b51fec.slice/crio-bf4248d05791d47248030924253ecbd8fdad4569eedc56630c85abc424bdf52c WatchSource:0}: Error finding container bf4248d05791d47248030924253ecbd8fdad4569eedc56630c85abc424bdf52c: Status 404 returned error can't find the container with id bf4248d05791d47248030924253ecbd8fdad4569eedc56630c85abc424bdf52c
Oct 30 20:53:17.075586 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:17.075565    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"]
Oct 30 20:53:17.078541 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:17.078507    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:53:17.078645 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:17.078580    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:53:17.078645 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:17.078615    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:53:17.931752 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:17.931710    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt" event={"ID":"d204447d-7147-415b-abf8-853f68b51fec","Type":"ContainerStarted","Data":"7da7aee651df96f9cb6a880e8f7d572a730af37552f32977d48c2c16a6c2edaa"}
Oct 30 20:53:17.932255 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:17.931768    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt" event={"ID":"d204447d-7147-415b-abf8-853f68b51fec","Type":"ContainerStarted","Data":"bf4248d05791d47248030924253ecbd8fdad4569eedc56630c85abc424bdf52c"}
Oct 30 20:53:18.102106 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:18.102048    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt" podStartSLOduration=3.10203317 podStartE2EDuration="3.10203317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:18.099451165 +0000 UTC m=+1610.016472700" watchObservedRunningTime="2026-10-30 20:53:18.10203317 +0000 UTC m=+1610.019054706"
Oct 30 20:53:18.519046 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:18.519011    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:53:18.524501 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:18.524479    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:53:18.940478 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:18.940452    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:53:18.941545 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:18.941520    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:53:21.011399 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:21.011311    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:21.230094 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:21.215219    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:21.402739 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:21.402694    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:23.821901 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:23.821871    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:24.002061 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:24.002007    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:24.312471 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:24.312432    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:24.562048 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:24.555793    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:26.539148 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:26.539105    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:31.406937 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:31.406900    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:31.798501 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:31.798469    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:32.660697 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:32.660659    5071 scope.go:122] "RemoveContainer" containerID="9dd405ea783d5a5ccf6243ab86b5f44f88e2da9d369b0352984fd04c2cfdcf06"
Oct 30 20:53:32.674382 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:32.674355    5071 scope.go:122] "RemoveContainer" containerID="42ce92727881463606893e301b3dccd7fb0b9102d0b52fd79d6f7d96a9d3d378"
Oct 30 20:53:32.685514 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:32.685469    5071 scope.go:122] "RemoveContainer" containerID="f7319e70e47d2bb6c1d7e4d5041f3aeb1a6de503d637a057665a66acb7dc687e"
Oct 30 20:53:32.696280 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:32.696262    5071 scope.go:122] "RemoveContainer" containerID="53ef8f47c3b223fd57e3f20d6b41237537d18d596752efd5042718c9ae17f1e3"
Oct 30 20:53:32.706068 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:32.706050    5071 scope.go:122] "RemoveContainer" containerID="acd97f9211a912240d1a112f3b74fd09e99b5eb88a7b9cf7e867fc6db7714d9b"
Oct 30 20:53:32.715743 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:32.715725    5071 scope.go:122] "RemoveContainer" containerID="a64ab1a5aafdc217552106d41bd42ffbf5e5dde053859890050e04011ce7663f"
Oct 30 20:53:32.726101 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:32.726078    5071 scope.go:122] "RemoveContainer" containerID="a4092b4b27794dea1811f53e63625ed1506201759dc5b277f6af9a6bb5df6634"
Oct 30 20:53:32.738611 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:32.738558    5071 scope.go:122] "RemoveContainer" containerID="05249dde4b9db70a45776c3fc32d921cea79b82af00228c84fef0a686152725d"
Oct 30 20:53:32.751981 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:32.751957    5071 scope.go:122] "RemoveContainer" containerID="c85eeac275a3bd5ac21b137dc48f73f726140ebb8b51c2c2ad18c2a21ac99af3"
Oct 30 20:53:32.764987 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:32.764957    5071 scope.go:122] "RemoveContainer" containerID="0ce2f6c6426d841e667dcba8348b3280a07d98db30bf06555f8f345c1f7deb82"
Oct 30 20:53:33.287847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:33.278344    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:33.502863 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:33.501954    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:33.635043 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:33.634955    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:53:35.156562 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:35.156522    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:35.654843 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:35.653520    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:36.088921 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:36.088142    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa138efb-67b2j"]
Oct 30 20:53:36.153688 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:36.153650    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa138efb-67b2j"]
Oct 30 20:53:36.283637 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:36.283600    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b913b363-6fbf-4031-a833-2e5e71ef1f19" path="/var/lib/kubelet/pods/b913b363-6fbf-4031-a833-2e5e71ef1f19/volumes"
Oct 30 20:53:36.349899 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:36.349721    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:37.497739 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:37.496844    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:53:37.723850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:37.717490    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:37.748690 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:37.748597    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:53:37.895309 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:37.895248    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:53:37.900987 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:37.900934    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:53:37.914681 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:37.911939    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 20:53:37.914681 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:37.913100    5071 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:53:38.153845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:38.153602    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:38.509330 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:38.509219    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:53:38.667779 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:38.667745    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:38.753006 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:38.752951    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:53:38.780902 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:38.778437    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:53:38.780902 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:38.778501    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:53:38.886116 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:38.886062    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:38.921866 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:38.917977    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 30 20:53:38.921866 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:38.918595    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:53:39.108853 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:39.105994    5071 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:53:39.427540 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:39.427456    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:53:39.574776 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:39.574732    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 20:53:39.735374 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:39.734672    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:53:39.919846 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:39.919164    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 20:53:39.919846 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:39.918957    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:53:40.040913 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:40.038719    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:53:40.460440 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:40.448358    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:53:40.720993 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:40.720895    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:53:40.820846 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:40.805416    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 20:53:43.194364 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:43.194329    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:43.226874 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:43.226761    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"]
Oct 30 20:53:43.468636 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:43.468595    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:45.170448 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:45.170366    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:46.471972 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:46.471933    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:46.827033 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:46.826559    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:47.059097 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:53:47.059050    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:53:47.143208 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:47.143116    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aaf669-656d468487-p5q8l"]
Oct 30 20:53:47.523932 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:47.522798    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aaf669-656d468487-p5q8l"]
Oct 30 20:53:47.523932 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:47.522880    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-aaf669-5994948fb8-5tw77"]
Oct 30 20:53:47.592952 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:47.525264    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aaf669-656d468487-p5q8l"
Oct 30 20:53:47.623007 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:47.615662    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-aaf669\""
Oct 30 20:53:47.623007 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:47.616470    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-aaf669-dockercfg-4gnqf\""
Oct 30 20:53:47.798891 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:47.789635    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147e612e-f84b-4696-ade0-bb3ba142dd4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-aaf669-656d468487-p5q8l\" (UID: \"147e612e-f84b-4696-ade0-bb3ba142dd4a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aaf669-656d468487-p5q8l"
Oct 30 20:53:47.798891 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:47.789733    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thqkd\" (UniqueName: \"kubernetes.io/projected/147e612e-f84b-4696-ade0-bb3ba142dd4a-kube-api-access-thqkd\") pod \"payload-pre-processing-e2e-shared-b-w2-aaf669-656d468487-p5q8l\" (UID: \"147e612e-f84b-4696-ade0-bb3ba142dd4a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aaf669-656d468487-p5q8l"
Oct 30 20:53:47.966849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:47.829363    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"]
Oct 30 20:53:47.966849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:47.829422    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"]
Oct 30 20:53:47.966849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:47.829455    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-aaf669-5994948fb8-5tw77"]
Oct 30 20:53:47.966849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:47.829666    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv" podUID="4349a44e-61fd-4a25-8548-96b11c3cf4cc" containerName="istio-proxy" containerID="cri-o://1acec5824a62533f0c9761f48fdd96ce155bf33284a9d7e594da98da2dead2d8" gracePeriod=30
Oct 30 20:53:47.966849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:47.829886    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-aaf669-5994948fb8-5tw77"
Oct 30 20:53:47.966849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:47.833184    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5" podUID="55a2c070-6473-481d-a202-f9fcd1231856" containerName="istio-proxy" containerID="cri-o://6ce7e12ba11a676a9836dafd1ab2e7c40725aa5f15ca6adf455ddfdb75720e1c" gracePeriod=30
Oct 30 20:53:47.966849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:47.842614    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"]
Oct 30 20:53:47.966849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:47.842891    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874" podUID="98829221-3804-4f1f-9abe-24101fb2a2a0" containerName="istio-proxy" containerID="cri-o://06eac27309063b9e1c9d873460dd1f9edd4732e62a78372e90a7b19c945cff74" gracePeriod=30
Oct 30 20:53:47.966849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:47.958387    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cce1c70d-cedf-494d-9abf-202a30a44263-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-aaf669-5994948fb8-5tw77\" (UID: \"cce1c70d-cedf-494d-9abf-202a30a44263\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-aaf669-5994948fb8-5tw77"
Oct 30 20:53:47.966849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:47.958496    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147e612e-f84b-4696-ade0-bb3ba142dd4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-aaf669-656d468487-p5q8l\" (UID: \"147e612e-f84b-4696-ade0-bb3ba142dd4a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aaf669-656d468487-p5q8l"
Oct 30 20:53:47.966849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:47.958592    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thqkd\" (UniqueName: \"kubernetes.io/projected/147e612e-f84b-4696-ade0-bb3ba142dd4a-kube-api-access-thqkd\") pod \"payload-pre-processing-e2e-shared-b-w2-aaf669-656d468487-p5q8l\" (UID: \"147e612e-f84b-4696-ade0-bb3ba142dd4a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aaf669-656d468487-p5q8l"
Oct 30 20:53:47.966849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:47.958733    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltklg\" (UniqueName: \"kubernetes.io/projected/cce1c70d-cedf-494d-9abf-202a30a44263-kube-api-access-ltklg\") pod \"payload-processing-e2e-shared-b-w2-aaf669-5994948fb8-5tw77\" (UID: \"cce1c70d-cedf-494d-9abf-202a30a44263\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-aaf669-5994948fb8-5tw77"
Oct 30 20:53:47.966849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:47.960158    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147e612e-f84b-4696-ade0-bb3ba142dd4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-aaf669-656d468487-p5q8l\" (UID: \"147e612e-f84b-4696-ade0-bb3ba142dd4a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aaf669-656d468487-p5q8l"
Oct 30 20:53:48.065174 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:48.065087    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltklg\" (UniqueName: \"kubernetes.io/projected/cce1c70d-cedf-494d-9abf-202a30a44263-kube-api-access-ltklg\") pod \"payload-processing-e2e-shared-b-w2-aaf669-5994948fb8-5tw77\" (UID: \"cce1c70d-cedf-494d-9abf-202a30a44263\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-aaf669-5994948fb8-5tw77"
Oct 30 20:53:48.067628 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:48.067549    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cce1c70d-cedf-494d-9abf-202a30a44263-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-aaf669-5994948fb8-5tw77\" (UID: \"cce1c70d-cedf-494d-9abf-202a30a44263\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-aaf669-5994948fb8-5tw77"
Oct 30 20:53:48.070401 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:48.070371    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cce1c70d-cedf-494d-9abf-202a30a44263-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-aaf669-5994948fb8-5tw77\" (UID: \"cce1c70d-cedf-494d-9abf-202a30a44263\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-aaf669-5994948fb8-5tw77"
Oct 30 20:53:48.082291 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:48.081960    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thqkd\" (UniqueName: \"kubernetes.io/projected/147e612e-f84b-4696-ade0-bb3ba142dd4a-kube-api-access-thqkd\") pod \"payload-pre-processing-e2e-shared-b-w2-aaf669-656d468487-p5q8l\" (UID: \"147e612e-f84b-4696-ade0-bb3ba142dd4a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aaf669-656d468487-p5q8l"
Oct 30 20:53:48.116961 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:48.116372    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"]
Oct 30 20:53:48.116961 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:48.116607    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz" podUID="203f757a-7115-466c-8a13-14fb326599ca" containerName="istio-proxy" containerID="cri-o://bdbe5346a4464e79bf2af7ce1592376856c73ec10df12e85f0b66ddb403ab5cc" gracePeriod=30
Oct 30 20:53:48.124185 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:48.124136    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"]
Oct 30 20:53:48.130585 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:48.130071    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp" podUID="cefe43d8-48c1-4ba9-923c-3c2da5db8f01" containerName="istio-proxy" containerID="cri-o://73abc8b1e0fe5fc25518ba3aa5acec46892d19c98ee6ee9a929d078bd526ba7c" gracePeriod=30
Oct 30 20:53:48.133264 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:48.132945    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"]
Oct 30 20:53:48.133264 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:48.133177    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq" podUID="3dbc434f-60ea-4a24-9115-b5db756cbec2" containerName="istio-proxy" containerID="cri-o://969c705a753fac45f41fb6ae2a8c853b62afd3c595a31498994a0428241d913f" gracePeriod=30
Oct 30 20:53:48.146567 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:48.146533    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltklg\" (UniqueName: \"kubernetes.io/projected/cce1c70d-cedf-494d-9abf-202a30a44263-kube-api-access-ltklg\") pod \"payload-processing-e2e-shared-b-w2-aaf669-5994948fb8-5tw77\" (UID: \"cce1c70d-cedf-494d-9abf-202a30a44263\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-aaf669-5994948fb8-5tw77"
Oct 30 20:53:48.147065 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:48.147039    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aaf669-656d468487-p5q8l"
Oct 30 20:53:48.160542 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:48.160303    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-aaf669-5994948fb8-5tw77"
Oct 30 20:53:48.576398 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:48.576187    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aaf669-656d468487-p5q8l"]
Oct 30 20:53:48.580593 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:53:48.580563    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod147e612e_f84b_4696_ade0_bb3ba142dd4a.slice/crio-1449c76aa6ab084d669f73369cf099c3f2dac5196e9bf66bb18de7273b6eb2fd WatchSource:0}: Error finding container 1449c76aa6ab084d669f73369cf099c3f2dac5196e9bf66bb18de7273b6eb2fd: Status 404 returned error can't find the container with id 1449c76aa6ab084d669f73369cf099c3f2dac5196e9bf66bb18de7273b6eb2fd
Oct 30 20:53:48.622118 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:53:48.622087    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce1c70d_cedf_494d_9abf_202a30a44263.slice/crio-c0555e369fe47e089e7ecd6483f3fdd81a200c232bff0f785973b56ef44826fb WatchSource:0}: Error finding container c0555e369fe47e089e7ecd6483f3fdd81a200c232bff0f785973b56ef44826fb: Status 404 returned error can't find the container with id c0555e369fe47e089e7ecd6483f3fdd81a200c232bff0f785973b56ef44826fb
Oct 30 20:53:48.623200 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:48.623177    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-aaf669-5994948fb8-5tw77"]
Oct 30 20:53:49.022220 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:49.022122    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:49.134530 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:49.134492    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aaf669-656d468487-p5q8l" event={"ID":"147e612e-f84b-4696-ade0-bb3ba142dd4a","Type":"ContainerStarted","Data":"611c7f32d3fd30e4e97701beadb28b180545014839ae53f3e525f2b94d2c94bc"}
Oct 30 20:53:49.134730 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:49.134557    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aaf669-656d468487-p5q8l" event={"ID":"147e612e-f84b-4696-ade0-bb3ba142dd4a","Type":"ContainerStarted","Data":"1449c76aa6ab084d669f73369cf099c3f2dac5196e9bf66bb18de7273b6eb2fd"}
Oct 30 20:53:49.134730 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:49.134584    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aaf669-656d468487-p5q8l"
Oct 30 20:53:49.136511 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:49.136485    5071 generic.go:356] "Generic (PLEG): container finished" podID="690bc007-90bc-4e66-8c45-6425aaccc813" containerID="fbba44203c5c4c37bf690ce7c7fe56f51c0c02777ad9ffb1ecad859af1f262a2" exitCode=0
Oct 30 20:53:49.136648 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:49.136565    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc" event={"ID":"690bc007-90bc-4e66-8c45-6425aaccc813","Type":"ContainerDied","Data":"fbba44203c5c4c37bf690ce7c7fe56f51c0c02777ad9ffb1ecad859af1f262a2"}
Oct 30 20:53:49.137130 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:49.137107    5071 scope.go:122] "RemoveContainer" containerID="fbba44203c5c4c37bf690ce7c7fe56f51c0c02777ad9ffb1ecad859af1f262a2"
Oct 30 20:53:49.138371 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:49.138345    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-aaf669-5994948fb8-5tw77" event={"ID":"cce1c70d-cedf-494d-9abf-202a30a44263","Type":"ContainerStarted","Data":"e93391e93283ae81ca8ff2785493335e08544d52f8cf775a6e7af1369c548f68"}
Oct 30 20:53:49.138465 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:49.138394    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-aaf669-5994948fb8-5tw77" event={"ID":"cce1c70d-cedf-494d-9abf-202a30a44263","Type":"ContainerStarted","Data":"c0555e369fe47e089e7ecd6483f3fdd81a200c232bff0f785973b56ef44826fb"}
Oct 30 20:53:49.138465 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:49.138424    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-aaf669-5994948fb8-5tw77"
Oct 30 20:53:49.139126 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:49.139100    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:53:49.139178 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:49.139166    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:53:49.139218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:49.139199    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:53:49.252517 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:49.252455    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aaf669-656d468487-p5q8l" podStartSLOduration=3.252441798 podStartE2EDuration="3.252441798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:49.251452611 +0000 UTC m=+1641.168474159" watchObservedRunningTime="2026-10-30 20:53:49.252441798 +0000 UTC m=+1641.169463348"
Oct 30 20:53:49.293486 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:49.293452    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:53:49.373711 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:49.373646    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-aaf669-5994948fb8-5tw77" podStartSLOduration=3.37362558 podStartE2EDuration="3.37362558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:49.367252535 +0000 UTC m=+1641.284274072" watchObservedRunningTime="2026-10-30 20:53:49.37362558 +0000 UTC m=+1641.290647118"
Oct 30 20:53:50.184589 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:50.184499    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc" event={"ID":"690bc007-90bc-4e66-8c45-6425aaccc813","Type":"ContainerStarted","Data":"44e87a29af5b1c144a5b4e7a219f53c313a785d9b1606a09289212b698776052"}
Oct 30 20:53:50.185309 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:50.185280    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b32941-66c49c7d84-pj6jk"]
Oct 30 20:53:50.306846 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:50.306779    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b32941-57bff4bc8c-qgdg8"]
Oct 30 20:53:50.598854 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:50.598731    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:50.658845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:50.653188    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:53:50.681601 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:50.681151    5071 patch_prober.go:29] interesting pod/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 20:53:50.681601 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:50.681210    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc" podUID="690bc007-90bc-4e66-8c45-6425aaccc813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 20:53:50.833990 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:50.822926    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:51.085858 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:51.075343    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:51.426895 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:51.416467    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:51.652850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:51.651359    5071 patch_prober.go:29] interesting pod/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 20:53:51.652850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:51.651438    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc" podUID="690bc007-90bc-4e66-8c45-6425aaccc813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 20:53:52.655846 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:52.652927    5071 patch_prober.go:29] interesting pod/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 20:53:52.655846 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:52.652987    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc" podUID="690bc007-90bc-4e66-8c45-6425aaccc813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 20:53:52.910474 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:52.909963    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:53:53.391405 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.387042    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"
Oct 30 20:53:53.614203 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.610053    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4349a44e-61fd-4a25-8548-96b11c3cf4cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4349a44e-61fd-4a25-8548-96b11c3cf4cc-istio-data\") pod \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\" (UID: \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\") "
Oct 30 20:53:53.614203 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.610114    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4349a44e-61fd-4a25-8548-96b11c3cf4cc-kube-api-access-fd27m\" (UniqueName: \"kubernetes.io/projected/4349a44e-61fd-4a25-8548-96b11c3cf4cc-kube-api-access-fd27m\") pod \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\" (UID: \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\") "
Oct 30 20:53:53.614203 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.610206    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4349a44e-61fd-4a25-8548-96b11c3cf4cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4349a44e-61fd-4a25-8548-96b11c3cf4cc-istio-envoy\") pod \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\" (UID: \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\") "
Oct 30 20:53:53.614203 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.610251    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4349a44e-61fd-4a25-8548-96b11c3cf4cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4349a44e-61fd-4a25-8548-96b11c3cf4cc-workload-certs\") pod \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\" (UID: \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\") "
Oct 30 20:53:53.614203 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.610296    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4349a44e-61fd-4a25-8548-96b11c3cf4cc-istio-token\" (UniqueName: \"kubernetes.io/projected/4349a44e-61fd-4a25-8548-96b11c3cf4cc-istio-token\") pod \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\" (UID: \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\") "
Oct 30 20:53:53.614203 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.610527    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4349a44e-61fd-4a25-8548-96b11c3cf4cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4349a44e-61fd-4a25-8548-96b11c3cf4cc-istio-podinfo\") pod \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\" (UID: \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\") "
Oct 30 20:53:53.614203 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.610680    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4349a44e-61fd-4a25-8548-96b11c3cf4cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4349a44e-61fd-4a25-8548-96b11c3cf4cc-workload-socket\") pod \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\" (UID: \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\") "
Oct 30 20:53:53.614203 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.610718    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4349a44e-61fd-4a25-8548-96b11c3cf4cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4349a44e-61fd-4a25-8548-96b11c3cf4cc-credential-socket\") pod \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\" (UID: \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\") "
Oct 30 20:53:53.614203 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.610869    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4349a44e-61fd-4a25-8548-96b11c3cf4cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4349a44e-61fd-4a25-8548-96b11c3cf4cc-istiod-ca-cert\") pod \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\" (UID: \"4349a44e-61fd-4a25-8548-96b11c3cf4cc\") "
Oct 30 20:53:53.623966 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.623886    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4349a44e-61fd-4a25-8548-96b11c3cf4cc-istiod-ca-cert" pod "4349a44e-61fd-4a25-8548-96b11c3cf4cc" (UID: "4349a44e-61fd-4a25-8548-96b11c3cf4cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:53:53.625843 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.625788    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4349a44e-61fd-4a25-8548-96b11c3cf4cc-workload-certs" pod "4349a44e-61fd-4a25-8548-96b11c3cf4cc" (UID: "4349a44e-61fd-4a25-8548-96b11c3cf4cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:53.674845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.674427    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4349a44e-61fd-4a25-8548-96b11c3cf4cc-istio-data" pod "4349a44e-61fd-4a25-8548-96b11c3cf4cc" (UID: "4349a44e-61fd-4a25-8548-96b11c3cf4cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:53.683018 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.680310    5071 patch_prober.go:29] interesting pod/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 20:53:53.683018 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.680366    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc" podUID="690bc007-90bc-4e66-8c45-6425aaccc813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 20:53:53.683018 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.680590    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4349a44e-61fd-4a25-8548-96b11c3cf4cc-credential-socket" pod "4349a44e-61fd-4a25-8548-96b11c3cf4cc" (UID: "4349a44e-61fd-4a25-8548-96b11c3cf4cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:53.683018 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.680869    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4349a44e-61fd-4a25-8548-96b11c3cf4cc-workload-socket" pod "4349a44e-61fd-4a25-8548-96b11c3cf4cc" (UID: "4349a44e-61fd-4a25-8548-96b11c3cf4cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:53.706517 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.706474    5071 generic.go:356] "Generic (PLEG): container finished" podID="203f757a-7115-466c-8a13-14fb326599ca" containerID="bdbe5346a4464e79bf2af7ce1592376856c73ec10df12e85f0b66ddb403ab5cc" exitCode=0
Oct 30 20:53:53.706692 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.706658    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz" event={"ID":"203f757a-7115-466c-8a13-14fb326599ca","Type":"ContainerDied","Data":"bdbe5346a4464e79bf2af7ce1592376856c73ec10df12e85f0b66ddb403ab5cc"}
Oct 30 20:53:53.712598 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.710695    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"
Oct 30 20:53:53.720451 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.717799    5071 generic.go:356] "Generic (PLEG): container finished" podID="55a2c070-6473-481d-a202-f9fcd1231856" containerID="6ce7e12ba11a676a9836dafd1ab2e7c40725aa5f15ca6adf455ddfdb75720e1c" exitCode=0
Oct 30 20:53:53.720451 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.717949    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5" event={"ID":"55a2c070-6473-481d-a202-f9fcd1231856","Type":"ContainerDied","Data":"6ce7e12ba11a676a9836dafd1ab2e7c40725aa5f15ca6adf455ddfdb75720e1c"}
Oct 30 20:53:53.720451 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.718265    5071 scope.go:122] "RemoveContainer" containerID="6ce7e12ba11a676a9836dafd1ab2e7c40725aa5f15ca6adf455ddfdb75720e1c"
Oct 30 20:53:53.720451 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.718617    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5" event={"ID":"55a2c070-6473-481d-a202-f9fcd1231856","Type":"ContainerDied","Data":"0b9ebdc57202f3a59a73a9bc8456cf75b866714f54e9e3420b1f59e637fdb24b"}
Oct 30 20:53:53.720451 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.720341    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4349a44e-61fd-4a25-8548-96b11c3cf4cc-istio-token" pod "4349a44e-61fd-4a25-8548-96b11c3cf4cc" (UID: "4349a44e-61fd-4a25-8548-96b11c3cf4cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:53.723606 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.721449    5071 generic.go:356] "Generic (PLEG): container finished" podID="cefe43d8-48c1-4ba9-923c-3c2da5db8f01" containerID="73abc8b1e0fe5fc25518ba3aa5acec46892d19c98ee6ee9a929d078bd526ba7c" exitCode=0
Oct 30 20:53:53.723606 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.721650    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp" event={"ID":"cefe43d8-48c1-4ba9-923c-3c2da5db8f01","Type":"ContainerDied","Data":"73abc8b1e0fe5fc25518ba3aa5acec46892d19c98ee6ee9a929d078bd526ba7c"}
Oct 30 20:53:53.724172 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.724064    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"
Oct 30 20:53:53.725385 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.725334    5071 generic.go:356] "Generic (PLEG): container finished" podID="4349a44e-61fd-4a25-8548-96b11c3cf4cc" containerID="1acec5824a62533f0c9761f48fdd96ce155bf33284a9d7e594da98da2dead2d8" exitCode=0
Oct 30 20:53:53.725816 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.725717    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv" event={"ID":"4349a44e-61fd-4a25-8548-96b11c3cf4cc","Type":"ContainerDied","Data":"1acec5824a62533f0c9761f48fdd96ce155bf33284a9d7e594da98da2dead2d8"}
Oct 30 20:53:53.725816 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.725781    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv" event={"ID":"4349a44e-61fd-4a25-8548-96b11c3cf4cc","Type":"ContainerDied","Data":"0132177a676b6cf677275ff3c267cc1e5b5d246a832fa0bf641535de02067048"}
Oct 30 20:53:53.726353 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.726260    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"
Oct 30 20:53:53.754986 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.736128    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4349a44e-61fd-4a25-8548-96b11c3cf4cc-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:53.754986 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.736161    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4349a44e-61fd-4a25-8548-96b11c3cf4cc-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:53.754986 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.736185    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4349a44e-61fd-4a25-8548-96b11c3cf4cc-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:53.754986 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.736201    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4349a44e-61fd-4a25-8548-96b11c3cf4cc-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:53.754986 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.736217    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4349a44e-61fd-4a25-8548-96b11c3cf4cc-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:53.754986 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.736232    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4349a44e-61fd-4a25-8548-96b11c3cf4cc-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:53.756997 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.756966    5071 generic.go:356] "Generic (PLEG): container finished" podID="3dbc434f-60ea-4a24-9115-b5db756cbec2" containerID="969c705a753fac45f41fb6ae2a8c853b62afd3c595a31498994a0428241d913f" exitCode=0
Oct 30 20:53:53.757342 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.757319    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq" event={"ID":"3dbc434f-60ea-4a24-9115-b5db756cbec2","Type":"ContainerDied","Data":"969c705a753fac45f41fb6ae2a8c853b62afd3c595a31498994a0428241d913f"}
Oct 30 20:53:53.759624 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.759598    5071 generic.go:356] "Generic (PLEG): container finished" podID="98829221-3804-4f1f-9abe-24101fb2a2a0" containerID="06eac27309063b9e1c9d873460dd1f9edd4732e62a78372e90a7b19c945cff74" exitCode=0
Oct 30 20:53:53.759800 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.759782    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874" event={"ID":"98829221-3804-4f1f-9abe-24101fb2a2a0","Type":"ContainerDied","Data":"06eac27309063b9e1c9d873460dd1f9edd4732e62a78372e90a7b19c945cff74"}
Oct 30 20:53:53.760062 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.760043    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"
Oct 30 20:53:53.800277 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.788377    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4349a44e-61fd-4a25-8548-96b11c3cf4cc-istio-envoy" pod "4349a44e-61fd-4a25-8548-96b11c3cf4cc" (UID: "4349a44e-61fd-4a25-8548-96b11c3cf4cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:53.800277 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.796057    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4349a44e-61fd-4a25-8548-96b11c3cf4cc-istio-podinfo" pod "4349a44e-61fd-4a25-8548-96b11c3cf4cc" (UID: "4349a44e-61fd-4a25-8548-96b11c3cf4cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:53:53.802153 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.800917    5071 scope.go:122] "RemoveContainer" containerID="6ce7e12ba11a676a9836dafd1ab2e7c40725aa5f15ca6adf455ddfdb75720e1c"
Oct 30 20:53:53.802153 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:53:53.801460    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ce7e12ba11a676a9836dafd1ab2e7c40725aa5f15ca6adf455ddfdb75720e1c\": container with ID starting with 6ce7e12ba11a676a9836dafd1ab2e7c40725aa5f15ca6adf455ddfdb75720e1c not found: ID does not exist" containerID="6ce7e12ba11a676a9836dafd1ab2e7c40725aa5f15ca6adf455ddfdb75720e1c"
Oct 30 20:53:53.802153 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.801492    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ce7e12ba11a676a9836dafd1ab2e7c40725aa5f15ca6adf455ddfdb75720e1c"} err="failed to get container status \"6ce7e12ba11a676a9836dafd1ab2e7c40725aa5f15ca6adf455ddfdb75720e1c\": rpc error: code = NotFound desc = could not find container \"6ce7e12ba11a676a9836dafd1ab2e7c40725aa5f15ca6adf455ddfdb75720e1c\": container with ID starting with 6ce7e12ba11a676a9836dafd1ab2e7c40725aa5f15ca6adf455ddfdb75720e1c not found: ID does not exist"
Oct 30 20:53:53.802153 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.801514    5071 scope.go:122] "RemoveContainer" containerID="1acec5824a62533f0c9761f48fdd96ce155bf33284a9d7e594da98da2dead2d8"
Oct 30 20:53:53.802153 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.802044    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4349a44e-61fd-4a25-8548-96b11c3cf4cc-kube-api-access-fd27m" pod "4349a44e-61fd-4a25-8548-96b11c3cf4cc" (UID: "4349a44e-61fd-4a25-8548-96b11c3cf4cc"). InnerVolumeSpecName "kube-api-access-fd27m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:53.817623 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.817532    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"
Oct 30 20:53:53.823739 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.823707    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"
Oct 30 20:53:53.824212 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.824195    5071 scope.go:122] "RemoveContainer" containerID="1acec5824a62533f0c9761f48fdd96ce155bf33284a9d7e594da98da2dead2d8"
Oct 30 20:53:53.824677 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:53:53.824646    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1acec5824a62533f0c9761f48fdd96ce155bf33284a9d7e594da98da2dead2d8\": container with ID starting with 1acec5824a62533f0c9761f48fdd96ce155bf33284a9d7e594da98da2dead2d8 not found: ID does not exist" containerID="1acec5824a62533f0c9761f48fdd96ce155bf33284a9d7e594da98da2dead2d8"
Oct 30 20:53:53.824783 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.824683    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1acec5824a62533f0c9761f48fdd96ce155bf33284a9d7e594da98da2dead2d8"} err="failed to get container status \"1acec5824a62533f0c9761f48fdd96ce155bf33284a9d7e594da98da2dead2d8\": rpc error: code = NotFound desc = could not find container \"1acec5824a62533f0c9761f48fdd96ce155bf33284a9d7e594da98da2dead2d8\": container with ID starting with 1acec5824a62533f0c9761f48fdd96ce155bf33284a9d7e594da98da2dead2d8 not found: ID does not exist"
Oct 30 20:53:53.824783 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.824699    5071 scope.go:122] "RemoveContainer" containerID="06eac27309063b9e1c9d873460dd1f9edd4732e62a78372e90a7b19c945cff74"
Oct 30 20:53:53.892167 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.892135    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dbc434f-60ea-4a24-9115-b5db756cbec2-istio-token\" (UniqueName: \"kubernetes.io/projected/3dbc434f-60ea-4a24-9115-b5db756cbec2-istio-token\") pod \"3dbc434f-60ea-4a24-9115-b5db756cbec2\" (UID: \"3dbc434f-60ea-4a24-9115-b5db756cbec2\") "
Oct 30 20:53:53.892347 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.892175    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-istiod-ca-cert\") pod \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\" (UID: \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\") "
Oct 30 20:53:53.892347 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.892202    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55a2c070-6473-481d-a202-f9fcd1231856-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55a2c070-6473-481d-a202-f9fcd1231856-istiod-ca-cert\") pod \"55a2c070-6473-481d-a202-f9fcd1231856\" (UID: \"55a2c070-6473-481d-a202-f9fcd1231856\") "
Oct 30 20:53:53.892347 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.892243    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dbc434f-60ea-4a24-9115-b5db756cbec2-kube-api-access-zf2vt\" (UniqueName: \"kubernetes.io/projected/3dbc434f-60ea-4a24-9115-b5db756cbec2-kube-api-access-zf2vt\") pod \"3dbc434f-60ea-4a24-9115-b5db756cbec2\" (UID: \"3dbc434f-60ea-4a24-9115-b5db756cbec2\") "
Oct 30 20:53:53.892347 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.892297    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98829221-3804-4f1f-9abe-24101fb2a2a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98829221-3804-4f1f-9abe-24101fb2a2a0-istio-data\") pod \"98829221-3804-4f1f-9abe-24101fb2a2a0\" (UID: \"98829221-3804-4f1f-9abe-24101fb2a2a0\") "
Oct 30 20:53:53.892664 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.892358    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dbc434f-60ea-4a24-9115-b5db756cbec2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbc434f-60ea-4a24-9115-b5db756cbec2-istiod-ca-cert\") pod \"3dbc434f-60ea-4a24-9115-b5db756cbec2\" (UID: \"3dbc434f-60ea-4a24-9115-b5db756cbec2\") "
Oct 30 20:53:53.892664 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.892401    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-istio-podinfo\") pod \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\" (UID: \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\") "
Oct 30 20:53:53.892664 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.892430    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-workload-certs\") pod \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\" (UID: \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\") "
Oct 30 20:53:53.892664 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.892494    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55a2c070-6473-481d-a202-f9fcd1231856-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55a2c070-6473-481d-a202-f9fcd1231856-istio-podinfo\") pod \"55a2c070-6473-481d-a202-f9fcd1231856\" (UID: \"55a2c070-6473-481d-a202-f9fcd1231856\") "
Oct 30 20:53:53.892664 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.892538    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98829221-3804-4f1f-9abe-24101fb2a2a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98829221-3804-4f1f-9abe-24101fb2a2a0-credential-socket\") pod \"98829221-3804-4f1f-9abe-24101fb2a2a0\" (UID: \"98829221-3804-4f1f-9abe-24101fb2a2a0\") "
Oct 30 20:53:53.892664 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.892566    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98829221-3804-4f1f-9abe-24101fb2a2a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98829221-3804-4f1f-9abe-24101fb2a2a0-istio-envoy\") pod \"98829221-3804-4f1f-9abe-24101fb2a2a0\" (UID: \"98829221-3804-4f1f-9abe-24101fb2a2a0\") "
Oct 30 20:53:53.892664 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.892600    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55a2c070-6473-481d-a202-f9fcd1231856-kube-api-access-k4v6n\" (UniqueName: \"kubernetes.io/projected/55a2c070-6473-481d-a202-f9fcd1231856-kube-api-access-k4v6n\") pod \"55a2c070-6473-481d-a202-f9fcd1231856\" (UID: \"55a2c070-6473-481d-a202-f9fcd1231856\") "
Oct 30 20:53:53.892664 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.892630    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98829221-3804-4f1f-9abe-24101fb2a2a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98829221-3804-4f1f-9abe-24101fb2a2a0-istiod-ca-cert\") pod \"98829221-3804-4f1f-9abe-24101fb2a2a0\" (UID: \"98829221-3804-4f1f-9abe-24101fb2a2a0\") "
Oct 30 20:53:53.893129 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.892667    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98829221-3804-4f1f-9abe-24101fb2a2a0-istio-token\" (UniqueName: \"kubernetes.io/projected/98829221-3804-4f1f-9abe-24101fb2a2a0-istio-token\") pod \"98829221-3804-4f1f-9abe-24101fb2a2a0\" (UID: \"98829221-3804-4f1f-9abe-24101fb2a2a0\") "
Oct 30 20:53:53.893129 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.892702    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55a2c070-6473-481d-a202-f9fcd1231856-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55a2c070-6473-481d-a202-f9fcd1231856-istio-data\") pod \"55a2c070-6473-481d-a202-f9fcd1231856\" (UID: \"55a2c070-6473-481d-a202-f9fcd1231856\") "
Oct 30 20:53:53.893129 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.892741    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbc434f-60ea-4a24-9115-b5db756cbec2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbc434f-60ea-4a24-9115-b5db756cbec2-workload-socket\") pod \"3dbc434f-60ea-4a24-9115-b5db756cbec2\" (UID: \"3dbc434f-60ea-4a24-9115-b5db756cbec2\") "
Oct 30 20:53:53.893129 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.892788    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbc434f-60ea-4a24-9115-b5db756cbec2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbc434f-60ea-4a24-9115-b5db756cbec2-credential-socket\") pod \"3dbc434f-60ea-4a24-9115-b5db756cbec2\" (UID: \"3dbc434f-60ea-4a24-9115-b5db756cbec2\") "
Oct 30 20:53:53.893129 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.892845    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98829221-3804-4f1f-9abe-24101fb2a2a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98829221-3804-4f1f-9abe-24101fb2a2a0-workload-socket\") pod \"98829221-3804-4f1f-9abe-24101fb2a2a0\" (UID: \"98829221-3804-4f1f-9abe-24101fb2a2a0\") "
Oct 30 20:53:53.893129 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.892874    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98829221-3804-4f1f-9abe-24101fb2a2a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98829221-3804-4f1f-9abe-24101fb2a2a0-workload-certs\") pod \"98829221-3804-4f1f-9abe-24101fb2a2a0\" (UID: \"98829221-3804-4f1f-9abe-24101fb2a2a0\") "
Oct 30 20:53:53.893129 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.892930    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-workload-socket\") pod \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\" (UID: \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\") "
Oct 30 20:53:53.893129 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.892964    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbc434f-60ea-4a24-9115-b5db756cbec2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbc434f-60ea-4a24-9115-b5db756cbec2-workload-certs\") pod \"3dbc434f-60ea-4a24-9115-b5db756cbec2\" (UID: \"3dbc434f-60ea-4a24-9115-b5db756cbec2\") "
Oct 30 20:53:53.893129 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.893014    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-istio-envoy\") pod \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\" (UID: \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\") "
Oct 30 20:53:53.893129 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.893106    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98829221-3804-4f1f-9abe-24101fb2a2a0-kube-api-access-ns95h\" (UniqueName: \"kubernetes.io/projected/98829221-3804-4f1f-9abe-24101fb2a2a0-kube-api-access-ns95h\") pod \"98829221-3804-4f1f-9abe-24101fb2a2a0\" (UID: \"98829221-3804-4f1f-9abe-24101fb2a2a0\") "
Oct 30 20:53:53.893634 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.893233    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3dbc434f-60ea-4a24-9115-b5db756cbec2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbc434f-60ea-4a24-9115-b5db756cbec2-istio-podinfo\") pod \"3dbc434f-60ea-4a24-9115-b5db756cbec2\" (UID: \"3dbc434f-60ea-4a24-9115-b5db756cbec2\") "
Oct 30 20:53:53.893634 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.893263    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55a2c070-6473-481d-a202-f9fcd1231856-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a2c070-6473-481d-a202-f9fcd1231856-workload-socket\") pod \"55a2c070-6473-481d-a202-f9fcd1231856\" (UID: \"55a2c070-6473-481d-a202-f9fcd1231856\") "
Oct 30 20:53:53.893634 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.893298    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-kube-api-access-k2g7d\" (UniqueName: \"kubernetes.io/projected/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-kube-api-access-k2g7d\") pod \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\" (UID: \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\") "
Oct 30 20:53:53.893634 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.893337    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55a2c070-6473-481d-a202-f9fcd1231856-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55a2c070-6473-481d-a202-f9fcd1231856-workload-certs\") pod \"55a2c070-6473-481d-a202-f9fcd1231856\" (UID: \"55a2c070-6473-481d-a202-f9fcd1231856\") "
Oct 30 20:53:53.893634 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.893367    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55a2c070-6473-481d-a202-f9fcd1231856-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a2c070-6473-481d-a202-f9fcd1231856-credential-socket\") pod \"55a2c070-6473-481d-a202-f9fcd1231856\" (UID: \"55a2c070-6473-481d-a202-f9fcd1231856\") "
Oct 30 20:53:53.893634 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.893403    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-istio-data\") pod \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\" (UID: \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\") "
Oct 30 20:53:53.893634 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.893442    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbc434f-60ea-4a24-9115-b5db756cbec2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbc434f-60ea-4a24-9115-b5db756cbec2-istio-envoy\") pod \"3dbc434f-60ea-4a24-9115-b5db756cbec2\" (UID: \"3dbc434f-60ea-4a24-9115-b5db756cbec2\") "
Oct 30 20:53:53.893634 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.893478    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-credential-socket\") pod \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\" (UID: \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\") "
Oct 30 20:53:53.893634 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.893550    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbc434f-60ea-4a24-9115-b5db756cbec2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbc434f-60ea-4a24-9115-b5db756cbec2-istio-data\") pod \"3dbc434f-60ea-4a24-9115-b5db756cbec2\" (UID: \"3dbc434f-60ea-4a24-9115-b5db756cbec2\") "
Oct 30 20:53:53.893634 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.893618    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-istio-token\" (UniqueName: \"kubernetes.io/projected/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-istio-token\") pod \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\" (UID: \"cefe43d8-48c1-4ba9-923c-3c2da5db8f01\") "
Oct 30 20:53:53.894199 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.893664    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55a2c070-6473-481d-a202-f9fcd1231856-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55a2c070-6473-481d-a202-f9fcd1231856-istio-envoy\") pod \"55a2c070-6473-481d-a202-f9fcd1231856\" (UID: \"55a2c070-6473-481d-a202-f9fcd1231856\") "
Oct 30 20:53:53.894199 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.893700    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98829221-3804-4f1f-9abe-24101fb2a2a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98829221-3804-4f1f-9abe-24101fb2a2a0-istio-podinfo\") pod \"98829221-3804-4f1f-9abe-24101fb2a2a0\" (UID: \"98829221-3804-4f1f-9abe-24101fb2a2a0\") "
Oct 30 20:53:53.894199 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.893756    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55a2c070-6473-481d-a202-f9fcd1231856-istio-token\" (UniqueName: \"kubernetes.io/projected/55a2c070-6473-481d-a202-f9fcd1231856-istio-token\") pod \"55a2c070-6473-481d-a202-f9fcd1231856\" (UID: \"55a2c070-6473-481d-a202-f9fcd1231856\") "
Oct 30 20:53:53.894664 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.894632    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd27m\" (UniqueName: \"kubernetes.io/projected/4349a44e-61fd-4a25-8548-96b11c3cf4cc-kube-api-access-fd27m\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:53.894740 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.894672    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4349a44e-61fd-4a25-8548-96b11c3cf4cc-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:53.894740 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.894687    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4349a44e-61fd-4a25-8548-96b11c3cf4cc-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:53.895458 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.895388    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98829221-3804-4f1f-9abe-24101fb2a2a0-workload-socket" pod "98829221-3804-4f1f-9abe-24101fb2a2a0" (UID: "98829221-3804-4f1f-9abe-24101fb2a2a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:53.896841 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.895472    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-workload-certs" pod "cefe43d8-48c1-4ba9-923c-3c2da5db8f01" (UID: "cefe43d8-48c1-4ba9-923c-3c2da5db8f01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:53.896973 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.895643    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98829221-3804-4f1f-9abe-24101fb2a2a0-istio-data" pod "98829221-3804-4f1f-9abe-24101fb2a2a0" (UID: "98829221-3804-4f1f-9abe-24101fb2a2a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:53.897303 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.897041    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dbc434f-60ea-4a24-9115-b5db756cbec2-istiod-ca-cert" pod "3dbc434f-60ea-4a24-9115-b5db756cbec2" (UID: "3dbc434f-60ea-4a24-9115-b5db756cbec2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:53:53.897303 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.896002    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98829221-3804-4f1f-9abe-24101fb2a2a0-istiod-ca-cert" pod "98829221-3804-4f1f-9abe-24101fb2a2a0" (UID: "98829221-3804-4f1f-9abe-24101fb2a2a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:53:53.897303 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.896227    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98829221-3804-4f1f-9abe-24101fb2a2a0-credential-socket" pod "98829221-3804-4f1f-9abe-24101fb2a2a0" (UID: "98829221-3804-4f1f-9abe-24101fb2a2a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:53.897727 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.897544    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbc434f-60ea-4a24-9115-b5db756cbec2-workload-socket" pod "3dbc434f-60ea-4a24-9115-b5db756cbec2" (UID: "3dbc434f-60ea-4a24-9115-b5db756cbec2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:53.898030 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.897869    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a2c070-6473-481d-a202-f9fcd1231856-istio-data" pod "55a2c070-6473-481d-a202-f9fcd1231856" (UID: "55a2c070-6473-481d-a202-f9fcd1231856"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:53.898369 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.898153    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbc434f-60ea-4a24-9115-b5db756cbec2-credential-socket" pod "3dbc434f-60ea-4a24-9115-b5db756cbec2" (UID: "3dbc434f-60ea-4a24-9115-b5db756cbec2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:53.899688 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.899345    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-istiod-ca-cert" pod "cefe43d8-48c1-4ba9-923c-3c2da5db8f01" (UID: "cefe43d8-48c1-4ba9-923c-3c2da5db8f01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:53:53.899688 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.899669    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55a2c070-6473-481d-a202-f9fcd1231856-istiod-ca-cert" pod "55a2c070-6473-481d-a202-f9fcd1231856" (UID: "55a2c070-6473-481d-a202-f9fcd1231856"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:53:53.899977 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.899948    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a2c070-6473-481d-a202-f9fcd1231856-workload-certs" pod "55a2c070-6473-481d-a202-f9fcd1231856" (UID: "55a2c070-6473-481d-a202-f9fcd1231856"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:53.900167 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.900145    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98829221-3804-4f1f-9abe-24101fb2a2a0-workload-certs" pod "98829221-3804-4f1f-9abe-24101fb2a2a0" (UID: "98829221-3804-4f1f-9abe-24101fb2a2a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:53.900382 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.900348    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-workload-socket" pod "cefe43d8-48c1-4ba9-923c-3c2da5db8f01" (UID: "cefe43d8-48c1-4ba9-923c-3c2da5db8f01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:53.900773 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.900751    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbc434f-60ea-4a24-9115-b5db756cbec2-workload-certs" pod "3dbc434f-60ea-4a24-9115-b5db756cbec2" (UID: "3dbc434f-60ea-4a24-9115-b5db756cbec2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:53.903629 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.903478    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a2c070-6473-481d-a202-f9fcd1231856-credential-socket" pod "55a2c070-6473-481d-a202-f9fcd1231856" (UID: "55a2c070-6473-481d-a202-f9fcd1231856"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:53.903839 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.903762    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-istio-data" pod "cefe43d8-48c1-4ba9-923c-3c2da5db8f01" (UID: "cefe43d8-48c1-4ba9-923c-3c2da5db8f01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:53.905651 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.905389    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-istio-envoy" pod "cefe43d8-48c1-4ba9-923c-3c2da5db8f01" (UID: "cefe43d8-48c1-4ba9-923c-3c2da5db8f01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:53.906937 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.906606    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55a2c070-6473-481d-a202-f9fcd1231856-istio-token" pod "55a2c070-6473-481d-a202-f9fcd1231856" (UID: "55a2c070-6473-481d-a202-f9fcd1231856"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:53.908451 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.908047    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98829221-3804-4f1f-9abe-24101fb2a2a0-istio-envoy" pod "98829221-3804-4f1f-9abe-24101fb2a2a0" (UID: "98829221-3804-4f1f-9abe-24101fb2a2a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:53.909166 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.908938    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbc434f-60ea-4a24-9115-b5db756cbec2-istio-envoy" pod "3dbc434f-60ea-4a24-9115-b5db756cbec2" (UID: "3dbc434f-60ea-4a24-9115-b5db756cbec2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:53.909557 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.909308    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbc434f-60ea-4a24-9115-b5db756cbec2-istio-data" pod "3dbc434f-60ea-4a24-9115-b5db756cbec2" (UID: "3dbc434f-60ea-4a24-9115-b5db756cbec2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:53.909557 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.909536    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-credential-socket" pod "cefe43d8-48c1-4ba9-923c-3c2da5db8f01" (UID: "cefe43d8-48c1-4ba9-923c-3c2da5db8f01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:53.909966 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.909600    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a2c070-6473-481d-a202-f9fcd1231856-workload-socket" pod "55a2c070-6473-481d-a202-f9fcd1231856" (UID: "55a2c070-6473-481d-a202-f9fcd1231856"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:53.915670 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.915243    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98829221-3804-4f1f-9abe-24101fb2a2a0-istio-podinfo" pod "98829221-3804-4f1f-9abe-24101fb2a2a0" (UID: "98829221-3804-4f1f-9abe-24101fb2a2a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:53:53.917178 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.917152    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-istio-token" pod "cefe43d8-48c1-4ba9-923c-3c2da5db8f01" (UID: "cefe43d8-48c1-4ba9-923c-3c2da5db8f01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:53.917986 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.917960    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3dbc434f-60ea-4a24-9115-b5db756cbec2-istio-podinfo" pod "3dbc434f-60ea-4a24-9115-b5db756cbec2" (UID: "3dbc434f-60ea-4a24-9115-b5db756cbec2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:53:53.918387 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.918364    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dbc434f-60ea-4a24-9115-b5db756cbec2-istio-token" pod "3dbc434f-60ea-4a24-9115-b5db756cbec2" (UID: "3dbc434f-60ea-4a24-9115-b5db756cbec2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:53.918601 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.918567    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dbc434f-60ea-4a24-9115-b5db756cbec2-kube-api-access-zf2vt" pod "3dbc434f-60ea-4a24-9115-b5db756cbec2" (UID: "3dbc434f-60ea-4a24-9115-b5db756cbec2"). InnerVolumeSpecName "kube-api-access-zf2vt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:53.918710 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.918603    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98829221-3804-4f1f-9abe-24101fb2a2a0-istio-token" pod "98829221-3804-4f1f-9abe-24101fb2a2a0" (UID: "98829221-3804-4f1f-9abe-24101fb2a2a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:53.918710 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.918677    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55a2c070-6473-481d-a202-f9fcd1231856-kube-api-access-k4v6n" pod "55a2c070-6473-481d-a202-f9fcd1231856" (UID: "55a2c070-6473-481d-a202-f9fcd1231856"). InnerVolumeSpecName "kube-api-access-k4v6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:53.918946 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.918926    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-istio-podinfo" pod "cefe43d8-48c1-4ba9-923c-3c2da5db8f01" (UID: "cefe43d8-48c1-4ba9-923c-3c2da5db8f01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:53:53.919276 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.919248    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55a2c070-6473-481d-a202-f9fcd1231856-istio-podinfo" pod "55a2c070-6473-481d-a202-f9fcd1231856" (UID: "55a2c070-6473-481d-a202-f9fcd1231856"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:53:53.919587 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.919542    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a2c070-6473-481d-a202-f9fcd1231856-istio-envoy" pod "55a2c070-6473-481d-a202-f9fcd1231856" (UID: "55a2c070-6473-481d-a202-f9fcd1231856"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:53.920143 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.920113    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-kube-api-access-k2g7d" pod "cefe43d8-48c1-4ba9-923c-3c2da5db8f01" (UID: "cefe43d8-48c1-4ba9-923c-3c2da5db8f01"). InnerVolumeSpecName "kube-api-access-k2g7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:53.922444 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:53.922410    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98829221-3804-4f1f-9abe-24101fb2a2a0-kube-api-access-ns95h" pod "98829221-3804-4f1f-9abe-24101fb2a2a0" (UID: "98829221-3804-4f1f-9abe-24101fb2a2a0"). InnerVolumeSpecName "kube-api-access-ns95h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:54.011165 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.010546    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98829221-3804-4f1f-9abe-24101fb2a2a0-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.011165 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.010578    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55a2c070-6473-481d-a202-f9fcd1231856-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.011165 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.010592    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dbc434f-60ea-4a24-9115-b5db756cbec2-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.011165 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.010605    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.011165 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.010618    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55a2c070-6473-481d-a202-f9fcd1231856-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.011165 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.010631    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf2vt\" (UniqueName: \"kubernetes.io/projected/3dbc434f-60ea-4a24-9115-b5db756cbec2-kube-api-access-zf2vt\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.011165 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.010656    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98829221-3804-4f1f-9abe-24101fb2a2a0-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.011165 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.010669    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbc434f-60ea-4a24-9115-b5db756cbec2-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.011165 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.010681    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.011165 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.010695    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.011165 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.010709    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55a2c070-6473-481d-a202-f9fcd1231856-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.011165 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.010723    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98829221-3804-4f1f-9abe-24101fb2a2a0-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.011165 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.010737    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98829221-3804-4f1f-9abe-24101fb2a2a0-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.011165 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.010753    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4v6n\" (UniqueName: \"kubernetes.io/projected/55a2c070-6473-481d-a202-f9fcd1231856-kube-api-access-k4v6n\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.011165 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.010766    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98829221-3804-4f1f-9abe-24101fb2a2a0-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.011165 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.010779    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98829221-3804-4f1f-9abe-24101fb2a2a0-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.011165 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.010793    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55a2c070-6473-481d-a202-f9fcd1231856-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.011165 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.010806    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbc434f-60ea-4a24-9115-b5db756cbec2-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.011165 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.011024    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbc434f-60ea-4a24-9115-b5db756cbec2-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.011165 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.011053    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98829221-3804-4f1f-9abe-24101fb2a2a0-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.011165 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.011068    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98829221-3804-4f1f-9abe-24101fb2a2a0-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.011165 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.011083    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.011165 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.011097    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbc434f-60ea-4a24-9115-b5db756cbec2-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.011165 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.011114    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.011165 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.011127    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns95h\" (UniqueName: \"kubernetes.io/projected/98829221-3804-4f1f-9abe-24101fb2a2a0-kube-api-access-ns95h\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.011165 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.011142    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbc434f-60ea-4a24-9115-b5db756cbec2-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.011165 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.011155    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a2c070-6473-481d-a202-f9fcd1231856-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.011165 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.011168    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2g7d\" (UniqueName: \"kubernetes.io/projected/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-kube-api-access-k2g7d\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.011165 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.011183    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55a2c070-6473-481d-a202-f9fcd1231856-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.014307 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.011196    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a2c070-6473-481d-a202-f9fcd1231856-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.014307 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.011209    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.014307 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.011222    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbc434f-60ea-4a24-9115-b5db756cbec2-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.014307 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.011237    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.014307 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.011251    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbc434f-60ea-4a24-9115-b5db756cbec2-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.014307 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.011264    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cefe43d8-48c1-4ba9-923c-3c2da5db8f01-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.014307 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.011278    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55a2c070-6473-481d-a202-f9fcd1231856-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.077139 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.077064    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"
Oct 30 20:53:54.112289 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.112258    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/203f757a-7115-466c-8a13-14fb326599ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/203f757a-7115-466c-8a13-14fb326599ca-istio-data\") pod \"203f757a-7115-466c-8a13-14fb326599ca\" (UID: \"203f757a-7115-466c-8a13-14fb326599ca\") "
Oct 30 20:53:54.112464 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.112331    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/203f757a-7115-466c-8a13-14fb326599ca-kube-api-access-lgl6v\" (UniqueName: \"kubernetes.io/projected/203f757a-7115-466c-8a13-14fb326599ca-kube-api-access-lgl6v\") pod \"203f757a-7115-466c-8a13-14fb326599ca\" (UID: \"203f757a-7115-466c-8a13-14fb326599ca\") "
Oct 30 20:53:54.112464 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.112356    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/203f757a-7115-466c-8a13-14fb326599ca-istio-token\" (UniqueName: \"kubernetes.io/projected/203f757a-7115-466c-8a13-14fb326599ca-istio-token\") pod \"203f757a-7115-466c-8a13-14fb326599ca\" (UID: \"203f757a-7115-466c-8a13-14fb326599ca\") "
Oct 30 20:53:54.112464 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.112427    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/203f757a-7115-466c-8a13-14fb326599ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/203f757a-7115-466c-8a13-14fb326599ca-istio-envoy\") pod \"203f757a-7115-466c-8a13-14fb326599ca\" (UID: \"203f757a-7115-466c-8a13-14fb326599ca\") "
Oct 30 20:53:54.112620 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.112578    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/203f757a-7115-466c-8a13-14fb326599ca-istio-data" pod "203f757a-7115-466c-8a13-14fb326599ca" (UID: "203f757a-7115-466c-8a13-14fb326599ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:54.112673 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.112638    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/203f757a-7115-466c-8a13-14fb326599ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/203f757a-7115-466c-8a13-14fb326599ca-credential-socket\") pod \"203f757a-7115-466c-8a13-14fb326599ca\" (UID: \"203f757a-7115-466c-8a13-14fb326599ca\") "
Oct 30 20:53:54.112790 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.112774    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/203f757a-7115-466c-8a13-14fb326599ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/203f757a-7115-466c-8a13-14fb326599ca-workload-certs\") pod \"203f757a-7115-466c-8a13-14fb326599ca\" (UID: \"203f757a-7115-466c-8a13-14fb326599ca\") "
Oct 30 20:53:54.112867 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.112815    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/203f757a-7115-466c-8a13-14fb326599ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/203f757a-7115-466c-8a13-14fb326599ca-workload-socket\") pod \"203f757a-7115-466c-8a13-14fb326599ca\" (UID: \"203f757a-7115-466c-8a13-14fb326599ca\") "
Oct 30 20:53:54.112920 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.112893    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/203f757a-7115-466c-8a13-14fb326599ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/203f757a-7115-466c-8a13-14fb326599ca-istio-podinfo\") pod \"203f757a-7115-466c-8a13-14fb326599ca\" (UID: \"203f757a-7115-466c-8a13-14fb326599ca\") "
Oct 30 20:53:54.112976 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.112936    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/203f757a-7115-466c-8a13-14fb326599ca-credential-socket" pod "203f757a-7115-466c-8a13-14fb326599ca" (UID: "203f757a-7115-466c-8a13-14fb326599ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:54.113029 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.112999    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/203f757a-7115-466c-8a13-14fb326599ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/203f757a-7115-466c-8a13-14fb326599ca-istiod-ca-cert\") pod \"203f757a-7115-466c-8a13-14fb326599ca\" (UID: \"203f757a-7115-466c-8a13-14fb326599ca\") "
Oct 30 20:53:54.113091 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.113075    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/203f757a-7115-466c-8a13-14fb326599ca-workload-certs" pod "203f757a-7115-466c-8a13-14fb326599ca" (UID: "203f757a-7115-466c-8a13-14fb326599ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:54.113222 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.113208    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/203f757a-7115-466c-8a13-14fb326599ca-workload-socket" pod "203f757a-7115-466c-8a13-14fb326599ca" (UID: "203f757a-7115-466c-8a13-14fb326599ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:54.113765 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.113742    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/203f757a-7115-466c-8a13-14fb326599ca-istiod-ca-cert" pod "203f757a-7115-466c-8a13-14fb326599ca" (UID: "203f757a-7115-466c-8a13-14fb326599ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:53:54.114041 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.114018    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/203f757a-7115-466c-8a13-14fb326599ca-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.114092 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.114050    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/203f757a-7115-466c-8a13-14fb326599ca-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.114092 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.114066    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/203f757a-7115-466c-8a13-14fb326599ca-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.114092 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.114080    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/203f757a-7115-466c-8a13-14fb326599ca-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.114190 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.114094    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/203f757a-7115-466c-8a13-14fb326599ca-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.115487 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.115465    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/203f757a-7115-466c-8a13-14fb326599ca-istio-token" pod "203f757a-7115-466c-8a13-14fb326599ca" (UID: "203f757a-7115-466c-8a13-14fb326599ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:54.115766 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.115740    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/203f757a-7115-466c-8a13-14fb326599ca-istio-envoy" pod "203f757a-7115-466c-8a13-14fb326599ca" (UID: "203f757a-7115-466c-8a13-14fb326599ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:54.116757 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.116732    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/203f757a-7115-466c-8a13-14fb326599ca-istio-podinfo" pod "203f757a-7115-466c-8a13-14fb326599ca" (UID: "203f757a-7115-466c-8a13-14fb326599ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:53:54.184018 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.183973    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/203f757a-7115-466c-8a13-14fb326599ca-kube-api-access-lgl6v" pod "203f757a-7115-466c-8a13-14fb326599ca" (UID: "203f757a-7115-466c-8a13-14fb326599ca"). InnerVolumeSpecName "kube-api-access-lgl6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:54.219908 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.219878    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgl6v\" (UniqueName: \"kubernetes.io/projected/203f757a-7115-466c-8a13-14fb326599ca-kube-api-access-lgl6v\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.220664 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.220633    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/203f757a-7115-466c-8a13-14fb326599ca-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.220847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.220789    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/203f757a-7115-466c-8a13-14fb326599ca-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.220847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.220810    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/203f757a-7115-466c-8a13-14fb326599ca-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:54.283524 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.283125    5071 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:53:54.283524 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.283194    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874" podUID="98829221-3804-4f1f-9abe-24101fb2a2a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:53:54.479444 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.479403    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"]
Oct 30 20:53:54.484049 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.484019    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-nq874"]
Oct 30 20:53:54.601032 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.601000    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"]
Oct 30 20:53:54.606877 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.606847    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-hp9fv"]
Oct 30 20:53:54.651846 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.651198    5071 patch_prober.go:29] interesting pod/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 20:53:54.651846 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.651267    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc" podUID="690bc007-90bc-4e66-8c45-6425aaccc813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 20:53:54.775683 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.775652    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq" event={"ID":"3dbc434f-60ea-4a24-9115-b5db756cbec2","Type":"ContainerDied","Data":"adc6a78935070e28c34685e75122f7f195f9d4e962b0adcce6949ca7c1bb5414"}
Oct 30 20:53:54.776222 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.775709    5071 scope.go:122] "RemoveContainer" containerID="969c705a753fac45f41fb6ae2a8c853b62afd3c595a31498994a0428241d913f"
Oct 30 20:53:54.776222 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.775781    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"
Oct 30 20:53:54.780549 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.780480    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz" event={"ID":"203f757a-7115-466c-8a13-14fb326599ca","Type":"ContainerDied","Data":"75324956feb95a2ef538374c564fe4ec246f9452d263e35b0e1b3c4ea6f6ae5a"}
Oct 30 20:53:54.780896 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.780868    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"
Oct 30 20:53:54.783215 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.783193    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"
Oct 30 20:53:54.785815 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.785798    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"
Oct 30 20:53:54.786321 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.785846    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp" event={"ID":"cefe43d8-48c1-4ba9-923c-3c2da5db8f01","Type":"ContainerDied","Data":"c92d22b26707cfd5bebebfba86cf1709beae8036b9bb0d124bf41aa1a9d0e39f"}
Oct 30 20:53:54.793562 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.793533    5071 scope.go:122] "RemoveContainer" containerID="bdbe5346a4464e79bf2af7ce1592376856c73ec10df12e85f0b66ddb403ab5cc"
Oct 30 20:53:54.804014 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.803993    5071 scope.go:122] "RemoveContainer" containerID="73abc8b1e0fe5fc25518ba3aa5acec46892d19c98ee6ee9a929d078bd526ba7c"
Oct 30 20:53:54.991391 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.991301    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"]
Oct 30 20:53:54.999800 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:54.999766    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-2pgs5"]
Oct 30 20:53:55.112733 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:55.112690    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"]
Oct 30 20:53:55.118481 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:55.118452    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-xp5zp"]
Oct 30 20:53:55.186246 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:55.186204    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-b32941-66c49c7d84-pj6jk" podUID="6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f" containerName="payload-processing" containerID="cri-o://47aa1564ddf759b7df0a648ac36dc75f1160382b129c0a3eece71243f10f06ca" gracePeriod=25
Oct 30 20:53:55.258115 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:55.258023    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"]
Oct 30 20:53:55.264838 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:55.264786    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-pwtrz"]
Oct 30 20:53:55.283481 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:55.283448    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ad0ff4fc-98f7856c8-mbzld"]
Oct 30 20:53:55.307909 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:55.307867    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b32941-57bff4bc8c-qgdg8" podUID="f218d322-bfbb-4396-8152-5b1138f476e8" containerName="payload-pre-processing" containerID="cri-o://52f1f49ab17276a0da6c8db5706a859d4b7d7112b3037bff95cb673509f7aa25" gracePeriod=25
Oct 30 20:53:55.360797 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:55.360756    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"]
Oct 30 20:53:55.365321 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:55.365296    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-h4ndq"]
Oct 30 20:53:55.410450 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:55.410418    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ad0ff4fc-768bd4dff9-lcr6k"]
Oct 30 20:53:55.659122 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:55.652430    5071 patch_prober.go:29] interesting pod/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 20:53:55.659501 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:55.659463    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc" podUID="690bc007-90bc-4e66-8c45-6425aaccc813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 20:53:55.688590 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:55.688554    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:56.004900 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:55.980404    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:56.271946 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:56.271799    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:56.425268 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:56.397322    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="203f757a-7115-466c-8a13-14fb326599ca" path="/var/lib/kubelet/pods/203f757a-7115-466c-8a13-14fb326599ca/volumes"
Oct 30 20:53:56.426713 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:56.426683    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dbc434f-60ea-4a24-9115-b5db756cbec2" path="/var/lib/kubelet/pods/3dbc434f-60ea-4a24-9115-b5db756cbec2/volumes"
Oct 30 20:53:56.427886 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:56.427868    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4349a44e-61fd-4a25-8548-96b11c3cf4cc" path="/var/lib/kubelet/pods/4349a44e-61fd-4a25-8548-96b11c3cf4cc/volumes"
Oct 30 20:53:56.434811 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:56.434791    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55a2c070-6473-481d-a202-f9fcd1231856" path="/var/lib/kubelet/pods/55a2c070-6473-481d-a202-f9fcd1231856/volumes"
Oct 30 20:53:56.463857 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:56.462234    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98829221-3804-4f1f-9abe-24101fb2a2a0" path="/var/lib/kubelet/pods/98829221-3804-4f1f-9abe-24101fb2a2a0/volumes"
Oct 30 20:53:56.463857 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:56.462935    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cefe43d8-48c1-4ba9-923c-3c2da5db8f01" path="/var/lib/kubelet/pods/cefe43d8-48c1-4ba9-923c-3c2da5db8f01/volumes"
Oct 30 20:53:56.648311 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:56.648268    5071 patch_prober.go:29] interesting pod/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 20:53:56.648506 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:56.648355    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc" podUID="690bc007-90bc-4e66-8c45-6425aaccc813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 20:53:57.465607 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:57.465569    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:57.678366 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:57.678334    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:53:57.680652 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:57.680624    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:53:57.688156 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:57.687352    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:53:58.255398 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:58.255355    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main" containerID="cri-o://2d151d574b82c2958e5dd9c5dde138f114fb177e3b62e42663244b57d101dd7a" gracePeriod=45
Oct 30 20:53:58.477475 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:58.477438    5071 generic.go:356] "Generic (PLEG): container finished" podID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerID="2d151d574b82c2958e5dd9c5dde138f114fb177e3b62e42663244b57d101dd7a" exitCode=0
Oct 30 20:53:58.478045 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:58.477978    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" event={"ID":"7ff8df1b-99b0-4b2d-979b-89946b9a16f6","Type":"ContainerDied","Data":"2d151d574b82c2958e5dd9c5dde138f114fb177e3b62e42663244b57d101dd7a"}
Oct 30 20:53:58.558080 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:58.558056    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:53:58.581945 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:58.581919    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-dshm\") pod \"7ff8df1b-99b0-4b2d-979b-89946b9a16f6\" (UID: \"7ff8df1b-99b0-4b2d-979b-89946b9a16f6\") "
Oct 30 20:53:58.582057 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:58.581970    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-kube-api-access-lcgdm\" (UniqueName: \"kubernetes.io/projected/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-kube-api-access-lcgdm\") pod \"7ff8df1b-99b0-4b2d-979b-89946b9a16f6\" (UID: \"7ff8df1b-99b0-4b2d-979b-89946b9a16f6\") "
Oct 30 20:53:58.582168 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:58.582157    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-home\" (UniqueName: \"kubernetes.io/empty-dir/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-home\") pod \"7ff8df1b-99b0-4b2d-979b-89946b9a16f6\" (UID: \"7ff8df1b-99b0-4b2d-979b-89946b9a16f6\") "
Oct 30 20:53:58.582204 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:58.582186    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-model-cache\") pod \"7ff8df1b-99b0-4b2d-979b-89946b9a16f6\" (UID: \"7ff8df1b-99b0-4b2d-979b-89946b9a16f6\") "
Oct 30 20:53:58.582267 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:58.582221    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-tmp-dir\") pod \"7ff8df1b-99b0-4b2d-979b-89946b9a16f6\" (UID: \"7ff8df1b-99b0-4b2d-979b-89946b9a16f6\") "
Oct 30 20:53:58.582331 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:58.582303    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-tls-certs\") pod \"7ff8df1b-99b0-4b2d-979b-89946b9a16f6\" (UID: \"7ff8df1b-99b0-4b2d-979b-89946b9a16f6\") "
Oct 30 20:53:58.582497 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:58.582467    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-model-cache" pod "7ff8df1b-99b0-4b2d-979b-89946b9a16f6" (UID: "7ff8df1b-99b0-4b2d-979b-89946b9a16f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:58.582497 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:58.582482    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-home" pod "7ff8df1b-99b0-4b2d-979b-89946b9a16f6" (UID: "7ff8df1b-99b0-4b2d-979b-89946b9a16f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:58.582958 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:58.582932    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-tmp-dir" pod "7ff8df1b-99b0-4b2d-979b-89946b9a16f6" (UID: "7ff8df1b-99b0-4b2d-979b-89946b9a16f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:58.583178 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:58.583160    5071 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-home\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:58.583233 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:58.583198    5071 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-model-cache\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:58.583233 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:58.583215    5071 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-tmp-dir\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:58.584731 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:58.584702    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-kube-api-access-lcgdm" pod "7ff8df1b-99b0-4b2d-979b-89946b9a16f6" (UID: "7ff8df1b-99b0-4b2d-979b-89946b9a16f6"). InnerVolumeSpecName "kube-api-access-lcgdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:58.584892 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:58.584776    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-dshm" pod "7ff8df1b-99b0-4b2d-979b-89946b9a16f6" (UID: "7ff8df1b-99b0-4b2d-979b-89946b9a16f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:58.585283 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:58.585254    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-tls-certs" pod "7ff8df1b-99b0-4b2d-979b-89946b9a16f6" (UID: "7ff8df1b-99b0-4b2d-979b-89946b9a16f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:53:58.684933 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:58.684896    5071 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-tls-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:58.684933 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:58.684929    5071 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-dshm\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:58.685191 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:58.684947    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcgdm\" (UniqueName: \"kubernetes.io/projected/7ff8df1b-99b0-4b2d-979b-89946b9a16f6-kube-api-access-lcgdm\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:59.380354 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:59.380322    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:59.764926 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:59.757578    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:53:59.784498 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:59.784468    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"
Oct 30 20:53:59.790032 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:59.789981    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4" event={"ID":"7ff8df1b-99b0-4b2d-979b-89946b9a16f6","Type":"ContainerDied","Data":"d3a369b928e69f7eb0840bc9725124418ab1f5f81f6af1b28a81146de4aebd78"}
Oct 30 20:53:59.790279 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:59.790242    5071 scope.go:122] "RemoveContainer" containerID="2d151d574b82c2958e5dd9c5dde138f114fb177e3b62e42663244b57d101dd7a"
Oct 30 20:53:59.838873 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:59.838815    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3d7c11-5d6df9947f-xrgfn"]
Oct 30 20:53:59.840559 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:59.840526    5071 cpu_manager.go:414] "containerMap: removing container" podUID="cefe43d8-48c1-4ba9-923c-3c2da5db8f01" containerName="istio-proxy"
Oct 30 20:53:59.840771 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:59.840756    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cefe43d8-48c1-4ba9-923c-3c2da5db8f01" containerName="istio-proxy"
Oct 30 20:53:59.840995 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:59.840980    5071 cpu_manager.go:414] "containerMap: removing container" podUID="55a2c070-6473-481d-a202-f9fcd1231856" containerName="istio-proxy"
Oct 30 20:53:59.841098 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:59.841084    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55a2c070-6473-481d-a202-f9fcd1231856" containerName="istio-proxy"
Oct 30 20:53:59.841216 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:59.841204    5071 cpu_manager.go:414] "containerMap: removing container" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main"
Oct 30 20:53:59.841318 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:59.841305    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main"
Oct 30 20:53:59.841412 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:59.841400    5071 cpu_manager.go:414] "containerMap: removing container" podUID="3dbc434f-60ea-4a24-9115-b5db756cbec2" containerName="istio-proxy"
Oct 30 20:53:59.841580 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:59.841566    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dbc434f-60ea-4a24-9115-b5db756cbec2" containerName="istio-proxy"
Oct 30 20:53:59.841861 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:59.841844    5071 cpu_manager.go:414] "containerMap: removing container" podUID="98829221-3804-4f1f-9abe-24101fb2a2a0" containerName="istio-proxy"
Oct 30 20:53:59.842030 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:59.841963    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98829221-3804-4f1f-9abe-24101fb2a2a0" containerName="istio-proxy"
Oct 30 20:53:59.842189 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:59.842174    5071 cpu_manager.go:414] "containerMap: removing container" podUID="203f757a-7115-466c-8a13-14fb326599ca" containerName="istio-proxy"
Oct 30 20:53:59.842288 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:59.842275    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="203f757a-7115-466c-8a13-14fb326599ca" containerName="istio-proxy"
Oct 30 20:53:59.842407 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:59.842395    5071 cpu_manager.go:414] "containerMap: removing container" podUID="4349a44e-61fd-4a25-8548-96b11c3cf4cc" containerName="istio-proxy"
Oct 30 20:53:59.842515 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:59.842501    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4349a44e-61fd-4a25-8548-96b11c3cf4cc" containerName="istio-proxy"
Oct 30 20:53:59.842834 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:59.842802    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="203f757a-7115-466c-8a13-14fb326599ca" containerName="istio-proxy"
Oct 30 20:53:59.842939 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:59.842926    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="4349a44e-61fd-4a25-8548-96b11c3cf4cc" containerName="istio-proxy"
Oct 30 20:53:59.843047 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:59.843037    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="cefe43d8-48c1-4ba9-923c-3c2da5db8f01" containerName="istio-proxy"
Oct 30 20:53:59.843803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:59.843786    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="55a2c070-6473-481d-a202-f9fcd1231856" containerName="istio-proxy"
Oct 30 20:53:59.843945 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:59.843933    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" containerName="main"
Oct 30 20:53:59.844022 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:59.844014    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dbc434f-60ea-4a24-9115-b5db756cbec2" containerName="istio-proxy"
Oct 30 20:53:59.844096 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:59.844088    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="98829221-3804-4f1f-9abe-24101fb2a2a0" containerName="istio-proxy"
Oct 30 20:53:59.911207 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:59.911178    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3d7c11-5d6df9947f-xrgfn"
Oct 30 20:53:59.982665 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:59.981543    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-3d7c11\""
Oct 30 20:53:59.982665 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:59.982345    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-3d7c11-dockercfg-96929\""
Oct 30 20:54:00.005047 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:59.999124    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pcnz\" (UniqueName: \"kubernetes.io/projected/ecaa3ce7-3945-4768-af31-1a0ea66825ee-kube-api-access-8pcnz\") pod \"payload-pre-processing-e2e-shared-a-w6-3d7c11-5d6df9947f-xrgfn\" (UID: \"ecaa3ce7-3945-4768-af31-1a0ea66825ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3d7c11-5d6df9947f-xrgfn"
Oct 30 20:54:00.005047 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:53:59.999333    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecaa3ce7-3945-4768-af31-1a0ea66825ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-3d7c11-5d6df9947f-xrgfn\" (UID: \"ecaa3ce7-3945-4768-af31-1a0ea66825ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3d7c11-5d6df9947f-xrgfn"
Oct 30 20:54:00.203183 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:00.134614    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecaa3ce7-3945-4768-af31-1a0ea66825ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-3d7c11-5d6df9947f-xrgfn\" (UID: \"ecaa3ce7-3945-4768-af31-1a0ea66825ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3d7c11-5d6df9947f-xrgfn"
Oct 30 20:54:00.203183 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:00.135667    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pcnz\" (UniqueName: \"kubernetes.io/projected/ecaa3ce7-3945-4768-af31-1a0ea66825ee-kube-api-access-8pcnz\") pod \"payload-pre-processing-e2e-shared-a-w6-3d7c11-5d6df9947f-xrgfn\" (UID: \"ecaa3ce7-3945-4768-af31-1a0ea66825ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3d7c11-5d6df9947f-xrgfn"
Oct 30 20:54:00.203183 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:00.136173    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecaa3ce7-3945-4768-af31-1a0ea66825ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-3d7c11-5d6df9947f-xrgfn\" (UID: \"ecaa3ce7-3945-4768-af31-1a0ea66825ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3d7c11-5d6df9947f-xrgfn"
Oct 30 20:54:00.266914 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:00.265620    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pcnz\" (UniqueName: \"kubernetes.io/projected/ecaa3ce7-3945-4768-af31-1a0ea66825ee-kube-api-access-8pcnz\") pod \"payload-pre-processing-e2e-shared-a-w6-3d7c11-5d6df9947f-xrgfn\" (UID: \"ecaa3ce7-3945-4768-af31-1a0ea66825ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3d7c11-5d6df9947f-xrgfn"
Oct 30 20:54:00.283805 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:00.283701    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ad0ff4fc-98f7856c8-mbzld" podUID="7947ba4c-bbde-45ff-a524-e00af39df6e6" containerName="payload-processing" containerID="cri-o://72e7aee37f4e036d1f662d63caf2474f91b8aa4695752aa39d0a027b44e6322a" gracePeriod=25
Oct 30 20:54:00.291976 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:00.291948    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aaf669-656d468487-p5q8l"
Oct 30 20:54:00.292244 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:00.292229    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-aaf669-5994948fb8-5tw77"
Oct 30 20:54:00.350711 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:00.350668    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3d7c11-5d6df9947f-xrgfn"]
Oct 30 20:54:00.371756 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:00.371721    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-3d7c11-698dd87b66-z282h"]
Oct 30 20:54:00.478872 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:00.411776    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad0ff4fc-768bd4dff9-lcr6k" podUID="07b0cd67-3dfd-47c0-b4d5-df6c3d61946b" containerName="payload-pre-processing" containerID="cri-o://4d855a5cbaec9919337530b32be1920f1d7c98c913e21e651663a9ae2578611e" gracePeriod=25
Oct 30 20:54:00.502700 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:00.502672    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-3d7c11-698dd87b66-z282h"]
Oct 30 20:54:00.503085 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:00.503067    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3d7c11-698dd87b66-z282h"
Oct 30 20:54:00.506526 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:00.506264    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3d7c11-5d6df9947f-xrgfn"
Oct 30 20:54:00.521933 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:00.521877    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"]
Oct 30 20:54:00.526778 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:00.526744    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b32941/simulator-w0-b32941-kserve-894b678fb-5gzn4"]
Oct 30 20:54:00.590990 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:00.590953    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6af92746-24d4-4bf5-b0b4-3840ace027cd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-3d7c11-698dd87b66-z282h\" (UID: \"6af92746-24d4-4bf5-b0b4-3840ace027cd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3d7c11-698dd87b66-z282h"
Oct 30 20:54:00.591422 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:00.591316    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9chm\" (UniqueName: \"kubernetes.io/projected/6af92746-24d4-4bf5-b0b4-3840ace027cd-kube-api-access-l9chm\") pod \"payload-processing-e2e-shared-a-w6-3d7c11-698dd87b66-z282h\" (UID: \"6af92746-24d4-4bf5-b0b4-3840ace027cd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3d7c11-698dd87b66-z282h"
Oct 30 20:54:00.710860 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:00.709848    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9chm\" (UniqueName: \"kubernetes.io/projected/6af92746-24d4-4bf5-b0b4-3840ace027cd-kube-api-access-l9chm\") pod \"payload-processing-e2e-shared-a-w6-3d7c11-698dd87b66-z282h\" (UID: \"6af92746-24d4-4bf5-b0b4-3840ace027cd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3d7c11-698dd87b66-z282h"
Oct 30 20:54:00.710860 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:00.710042    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6af92746-24d4-4bf5-b0b4-3840ace027cd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-3d7c11-698dd87b66-z282h\" (UID: \"6af92746-24d4-4bf5-b0b4-3840ace027cd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3d7c11-698dd87b66-z282h"
Oct 30 20:54:00.711483 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:00.711274    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6af92746-24d4-4bf5-b0b4-3840ace027cd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-3d7c11-698dd87b66-z282h\" (UID: \"6af92746-24d4-4bf5-b0b4-3840ace027cd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3d7c11-698dd87b66-z282h"
Oct 30 20:54:00.777353 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:00.777317    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9chm\" (UniqueName: \"kubernetes.io/projected/6af92746-24d4-4bf5-b0b4-3840ace027cd-kube-api-access-l9chm\") pod \"payload-processing-e2e-shared-a-w6-3d7c11-698dd87b66-z282h\" (UID: \"6af92746-24d4-4bf5-b0b4-3840ace027cd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3d7c11-698dd87b66-z282h"
Oct 30 20:54:00.862921 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:00.852936    5071 generic.go:356] "Generic (PLEG): container finished" podID="07b0cd67-3dfd-47c0-b4d5-df6c3d61946b" containerID="4d855a5cbaec9919337530b32be1920f1d7c98c913e21e651663a9ae2578611e" exitCode=0
Oct 30 20:54:00.862921 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:00.852994    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad0ff4fc-768bd4dff9-lcr6k" event={"ID":"07b0cd67-3dfd-47c0-b4d5-df6c3d61946b","Type":"ContainerDied","Data":"4d855a5cbaec9919337530b32be1920f1d7c98c913e21e651663a9ae2578611e"}
Oct 30 20:54:00.862921 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:00.857625    5071 generic.go:356] "Generic (PLEG): container finished" podID="7947ba4c-bbde-45ff-a524-e00af39df6e6" containerID="72e7aee37f4e036d1f662d63caf2474f91b8aa4695752aa39d0a027b44e6322a" exitCode=0
Oct 30 20:54:00.862921 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:00.857711    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ad0ff4fc-98f7856c8-mbzld" event={"ID":"7947ba4c-bbde-45ff-a524-e00af39df6e6","Type":"ContainerDied","Data":"72e7aee37f4e036d1f662d63caf2474f91b8aa4695752aa39d0a027b44e6322a"}
Oct 30 20:54:00.869738 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:00.869720    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ad0ff4fc-98f7856c8-mbzld"
Oct 30 20:54:00.906949 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:00.906917    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad0ff4fc-768bd4dff9-lcr6k"
Oct 30 20:54:00.908212 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:00.908190    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3d7c11-698dd87b66-z282h"
Oct 30 20:54:00.956041 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:54:00.956009    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecaa3ce7_3945_4768_af31_1a0ea66825ee.slice/crio-fb3dc52d09db3994651332d4ef4139d907513bdc58143d7dc231f726392fdb1f WatchSource:0}: Error finding container fb3dc52d09db3994651332d4ef4139d907513bdc58143d7dc231f726392fdb1f: Status 404 returned error can't find the container with id fb3dc52d09db3994651332d4ef4139d907513bdc58143d7dc231f726392fdb1f
Oct 30 20:54:00.956302 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:00.956279    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3d7c11-5d6df9947f-xrgfn"]
Oct 30 20:54:01.036673 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:01.036629    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7947ba4c-bbde-45ff-a524-e00af39df6e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7947ba4c-bbde-45ff-a524-e00af39df6e6-plugins-config-volume\") pod \"7947ba4c-bbde-45ff-a524-e00af39df6e6\" (UID: \"7947ba4c-bbde-45ff-a524-e00af39df6e6\") "
Oct 30 20:54:01.036887 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:01.036865    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7947ba4c-bbde-45ff-a524-e00af39df6e6-kube-api-access-dfk72\" (UniqueName: \"kubernetes.io/projected/7947ba4c-bbde-45ff-a524-e00af39df6e6-kube-api-access-dfk72\") pod \"7947ba4c-bbde-45ff-a524-e00af39df6e6\" (UID: \"7947ba4c-bbde-45ff-a524-e00af39df6e6\") "
Oct 30 20:54:01.036984 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:01.036966    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7947ba4c-bbde-45ff-a524-e00af39df6e6-plugins-config-volume" pod "7947ba4c-bbde-45ff-a524-e00af39df6e6" (UID: "7947ba4c-bbde-45ff-a524-e00af39df6e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:01.037147 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:01.037129    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07b0cd67-3dfd-47c0-b4d5-df6c3d61946b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b0cd67-3dfd-47c0-b4d5-df6c3d61946b-plugins-config-volume\") pod \"07b0cd67-3dfd-47c0-b4d5-df6c3d61946b\" (UID: \"07b0cd67-3dfd-47c0-b4d5-df6c3d61946b\") "
Oct 30 20:54:01.037331 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:01.037301    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07b0cd67-3dfd-47c0-b4d5-df6c3d61946b-kube-api-access-kn52l\" (UniqueName: \"kubernetes.io/projected/07b0cd67-3dfd-47c0-b4d5-df6c3d61946b-kube-api-access-kn52l\") pod \"07b0cd67-3dfd-47c0-b4d5-df6c3d61946b\" (UID: \"07b0cd67-3dfd-47c0-b4d5-df6c3d61946b\") "
Oct 30 20:54:01.037846 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:01.037804    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07b0cd67-3dfd-47c0-b4d5-df6c3d61946b-plugins-config-volume" pod "07b0cd67-3dfd-47c0-b4d5-df6c3d61946b" (UID: "07b0cd67-3dfd-47c0-b4d5-df6c3d61946b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:01.038285 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:01.038264    5071 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b0cd67-3dfd-47c0-b4d5-df6c3d61946b-plugins-config-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:01.038370 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:01.038304    5071 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7947ba4c-bbde-45ff-a524-e00af39df6e6-plugins-config-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:01.041447 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:01.041415    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7947ba4c-bbde-45ff-a524-e00af39df6e6-kube-api-access-dfk72" pod "7947ba4c-bbde-45ff-a524-e00af39df6e6" (UID: "7947ba4c-bbde-45ff-a524-e00af39df6e6"). InnerVolumeSpecName "kube-api-access-dfk72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:01.043531 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:01.043496    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07b0cd67-3dfd-47c0-b4d5-df6c3d61946b-kube-api-access-kn52l" pod "07b0cd67-3dfd-47c0-b4d5-df6c3d61946b" (UID: "07b0cd67-3dfd-47c0-b4d5-df6c3d61946b"). InnerVolumeSpecName "kube-api-access-kn52l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:01.140225 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:01.140129    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfk72\" (UniqueName: \"kubernetes.io/projected/7947ba4c-bbde-45ff-a524-e00af39df6e6-kube-api-access-dfk72\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:01.140225 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:01.140175    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn52l\" (UniqueName: \"kubernetes.io/projected/07b0cd67-3dfd-47c0-b4d5-df6c3d61946b-kube-api-access-kn52l\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:01.308761 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:54:01.308729    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6af92746_24d4_4bf5_b0b4_3840ace027cd.slice/crio-b22117f87837785292d5a1514d3c3bde7fcef8406ada48b8d807cb4d7c5834cc WatchSource:0}: Error finding container b22117f87837785292d5a1514d3c3bde7fcef8406ada48b8d807cb4d7c5834cc: Status 404 returned error can't find the container with id b22117f87837785292d5a1514d3c3bde7fcef8406ada48b8d807cb4d7c5834cc
Oct 30 20:54:01.308980 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:01.308946    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-3d7c11-698dd87b66-z282h"]
Oct 30 20:54:01.868016 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:01.867976    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3d7c11-698dd87b66-z282h" event={"ID":"6af92746-24d4-4bf5-b0b4-3840ace027cd","Type":"ContainerStarted","Data":"c2907229006b1f0cc980ebf57f327df53f50698fa5f54eaa57c31e2d82577d6a"}
Oct 30 20:54:01.868555 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:01.868040    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3d7c11-698dd87b66-z282h" event={"ID":"6af92746-24d4-4bf5-b0b4-3840ace027cd","Type":"ContainerStarted","Data":"b22117f87837785292d5a1514d3c3bde7fcef8406ada48b8d807cb4d7c5834cc"}
Oct 30 20:54:01.868555 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:01.868073    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3d7c11-698dd87b66-z282h"
Oct 30 20:54:01.869959 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:01.869932    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3d7c11-5d6df9947f-xrgfn" event={"ID":"ecaa3ce7-3945-4768-af31-1a0ea66825ee","Type":"ContainerStarted","Data":"b770efbc384b04447dd611a64201f0de3baa989ee9d967e4ef5677fe34e35f44"}
Oct 30 20:54:01.870093 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:01.869983    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3d7c11-5d6df9947f-xrgfn" event={"ID":"ecaa3ce7-3945-4768-af31-1a0ea66825ee","Type":"ContainerStarted","Data":"fb3dc52d09db3994651332d4ef4139d907513bdc58143d7dc231f726392fdb1f"}
Oct 30 20:54:01.870093 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:01.870056    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3d7c11-5d6df9947f-xrgfn"
Oct 30 20:54:01.871531 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:01.871511    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ad0ff4fc-98f7856c8-mbzld"
Oct 30 20:54:01.871633 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:01.871558    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ad0ff4fc-98f7856c8-mbzld" event={"ID":"7947ba4c-bbde-45ff-a524-e00af39df6e6","Type":"ContainerDied","Data":"cf4bb4d076ef1d1508faa199f5881102b7449e7de9f0469eb275a5461718954b"}
Oct 30 20:54:01.871633 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:01.871609    5071 scope.go:122] "RemoveContainer" containerID="72e7aee37f4e036d1f662d63caf2474f91b8aa4695752aa39d0a027b44e6322a"
Oct 30 20:54:01.873549 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:01.873517    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad0ff4fc-768bd4dff9-lcr6k"
Oct 30 20:54:01.873666 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:01.873555    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad0ff4fc-768bd4dff9-lcr6k" event={"ID":"07b0cd67-3dfd-47c0-b4d5-df6c3d61946b","Type":"ContainerDied","Data":"c441f37741e04e29b37ad6d12d15b9a2758fbc5fea42fddd3222af8f119c59e7"}
Oct 30 20:54:01.884039 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:01.884017    5071 scope.go:122] "RemoveContainer" containerID="4d855a5cbaec9919337530b32be1920f1d7c98c913e21e651663a9ae2578611e"
Oct 30 20:54:01.987087 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:01.987032    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3d7c11-698dd87b66-z282h" podStartSLOduration=2.987016043 podStartE2EDuration="2.987016043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:01.984534917 +0000 UTC m=+1653.901556453" watchObservedRunningTime="2026-10-30 20:54:01.987016043 +0000 UTC m=+1653.904037580"
Oct 30 20:54:02.101665 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:02.101610    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3d7c11-5d6df9947f-xrgfn" podStartSLOduration=3.101597342 podStartE2EDuration="3.101597342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:02.101252504 +0000 UTC m=+1654.018274039" watchObservedRunningTime="2026-10-30 20:54:02.101597342 +0000 UTC m=+1654.018618877"
Oct 30 20:54:02.217557 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:02.217473    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ff8df1b-99b0-4b2d-979b-89946b9a16f6" path="/var/lib/kubelet/pods/7ff8df1b-99b0-4b2d-979b-89946b9a16f6/volumes"
Oct 30 20:54:02.224145 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:02.224114    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"]
Oct 30 20:54:02.225963 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:02.225420    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8" podUID="191943aa-3324-4132-9edf-d2fbfc748b87" containerName="istio-proxy" containerID="cri-o://98f0c3af6a8bcfce29ea2314e71d8449f351240bbbb1d90cad38aeec82338608" gracePeriod=30
Oct 30 20:54:02.286073 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:02.286042    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"]
Oct 30 20:54:02.286627 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:02.286589    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk" podUID="6295120b-8bbc-4502-8429-4f04d7d5c5a4" containerName="istio-proxy" containerID="cri-o://45b8103cf080d88ad6ba8310fa33035ccf747e9b9195dc503291a1adba11ccc9" gracePeriod=30
Oct 30 20:54:02.316279 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:02.316247    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"]
Oct 30 20:54:02.316801 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:02.316768    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42" podUID="c4d634d9-39de-4614-9599-38adf86829e2" containerName="istio-proxy" containerID="cri-o://ccecb5607c37595ea3fe07271e177ea3db15296bbeecbed3def11b721934dda7" gracePeriod=30
Oct 30 20:54:02.335050 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:02.334291    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ad0ff4fc-768bd4dff9-lcr6k"]
Oct 30 20:54:02.388354 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:02.383729    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ad0ff4fc-768bd4dff9-lcr6k"]
Oct 30 20:54:02.428514 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:02.428473    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ad0ff4fc-98f7856c8-mbzld"]
Oct 30 20:54:02.436888 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:02.434867    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ad0ff4fc-98f7856c8-mbzld"]
Oct 30 20:54:03.187724 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:03.187688    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:03.640178 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:03.640131    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-b32941-66c49c7d84-pj6jk" podUID="6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:54:03.655429 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:03.655388    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b32941-57bff4bc8c-qgdg8" podUID="f218d322-bfbb-4396-8152-5b1138f476e8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:54:04.205073 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:04.205043    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07b0cd67-3dfd-47c0-b4d5-df6c3d61946b" path="/var/lib/kubelet/pods/07b0cd67-3dfd-47c0-b4d5-df6c3d61946b/volumes"
Oct 30 20:54:04.205477 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:04.205460    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7947ba4c-bbde-45ff-a524-e00af39df6e6" path="/var/lib/kubelet/pods/7947ba4c-bbde-45ff-a524-e00af39df6e6/volumes"
Oct 30 20:54:06.170498 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:06.170459    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:54:06.170953 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:06.170531    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk" podUID="6295120b-8bbc-4502-8429-4f04d7d5c5a4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:54:07.332540 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:07.332493    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:54:07.332985 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:07.332565    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42" podUID="c4d634d9-39de-4614-9599-38adf86829e2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:54:07.668423 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:07.668397    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"
Oct 30 20:54:07.921989 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:07.921880    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6295120b-8bbc-4502-8429-4f04d7d5c5a4-istio-token\" (UniqueName: \"kubernetes.io/projected/6295120b-8bbc-4502-8429-4f04d7d5c5a4-istio-token\") pod \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\" (UID: \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\") "
Oct 30 20:54:07.922328 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:07.922299    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6295120b-8bbc-4502-8429-4f04d7d5c5a4-kube-api-access-c82rd\" (UniqueName: \"kubernetes.io/projected/6295120b-8bbc-4502-8429-4f04d7d5c5a4-kube-api-access-c82rd\") pod \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\" (UID: \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\") "
Oct 30 20:54:07.930846 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:07.922579    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6295120b-8bbc-4502-8429-4f04d7d5c5a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6295120b-8bbc-4502-8429-4f04d7d5c5a4-workload-certs\") pod \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\" (UID: \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\") "
Oct 30 20:54:07.931664 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:07.931631    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6295120b-8bbc-4502-8429-4f04d7d5c5a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6295120b-8bbc-4502-8429-4f04d7d5c5a4-istio-envoy\") pod \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\" (UID: \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\") "
Oct 30 20:54:07.931939 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:07.931922    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6295120b-8bbc-4502-8429-4f04d7d5c5a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6295120b-8bbc-4502-8429-4f04d7d5c5a4-istio-podinfo\") pod \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\" (UID: \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\") "
Oct 30 20:54:07.932116 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:07.932102    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6295120b-8bbc-4502-8429-4f04d7d5c5a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6295120b-8bbc-4502-8429-4f04d7d5c5a4-workload-socket\") pod \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\" (UID: \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\") "
Oct 30 20:54:07.932214 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:07.932200    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6295120b-8bbc-4502-8429-4f04d7d5c5a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6295120b-8bbc-4502-8429-4f04d7d5c5a4-istiod-ca-cert\") pod \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\" (UID: \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\") "
Oct 30 20:54:07.932403 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:07.932373    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6295120b-8bbc-4502-8429-4f04d7d5c5a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6295120b-8bbc-4502-8429-4f04d7d5c5a4-credential-socket\") pod \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\" (UID: \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\") "
Oct 30 20:54:07.932609 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:07.932590    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6295120b-8bbc-4502-8429-4f04d7d5c5a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6295120b-8bbc-4502-8429-4f04d7d5c5a4-istio-data\") pod \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\" (UID: \"6295120b-8bbc-4502-8429-4f04d7d5c5a4\") "
Oct 30 20:54:07.957848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:07.947384    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6295120b-8bbc-4502-8429-4f04d7d5c5a4-workload-socket" pod "6295120b-8bbc-4502-8429-4f04d7d5c5a4" (UID: "6295120b-8bbc-4502-8429-4f04d7d5c5a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:07.995033 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:07.958349    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6295120b-8bbc-4502-8429-4f04d7d5c5a4-istiod-ca-cert" pod "6295120b-8bbc-4502-8429-4f04d7d5c5a4" (UID: "6295120b-8bbc-4502-8429-4f04d7d5c5a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:07.995033 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:07.964342    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6295120b-8bbc-4502-8429-4f04d7d5c5a4-workload-certs" pod "6295120b-8bbc-4502-8429-4f04d7d5c5a4" (UID: "6295120b-8bbc-4502-8429-4f04d7d5c5a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:07.995033 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:07.980334    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6295120b-8bbc-4502-8429-4f04d7d5c5a4-istio-token" pod "6295120b-8bbc-4502-8429-4f04d7d5c5a4" (UID: "6295120b-8bbc-4502-8429-4f04d7d5c5a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:07.996244 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:07.995435    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6295120b-8bbc-4502-8429-4f04d7d5c5a4-credential-socket" pod "6295120b-8bbc-4502-8429-4f04d7d5c5a4" (UID: "6295120b-8bbc-4502-8429-4f04d7d5c5a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:07.997066 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:07.997039    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6295120b-8bbc-4502-8429-4f04d7d5c5a4-istio-data" pod "6295120b-8bbc-4502-8429-4f04d7d5c5a4" (UID: "6295120b-8bbc-4502-8429-4f04d7d5c5a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:08.060009 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.036366    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6295120b-8bbc-4502-8429-4f04d7d5c5a4-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:08.121951 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.107875    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6295120b-8bbc-4502-8429-4f04d7d5c5a4-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:08.121951 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.108002    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6295120b-8bbc-4502-8429-4f04d7d5c5a4-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:08.121951 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.108019    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6295120b-8bbc-4502-8429-4f04d7d5c5a4-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:08.121951 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.108036    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6295120b-8bbc-4502-8429-4f04d7d5c5a4-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:08.121951 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.108050    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6295120b-8bbc-4502-8429-4f04d7d5c5a4-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:08.121951 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.071330    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6295120b-8bbc-4502-8429-4f04d7d5c5a4-kube-api-access-c82rd" pod "6295120b-8bbc-4502-8429-4f04d7d5c5a4" (UID: "6295120b-8bbc-4502-8429-4f04d7d5c5a4"). InnerVolumeSpecName "kube-api-access-c82rd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:08.161962 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.161925    5071 generic.go:356] "Generic (PLEG): container finished" podID="6295120b-8bbc-4502-8429-4f04d7d5c5a4" containerID="45b8103cf080d88ad6ba8310fa33035ccf747e9b9195dc503291a1adba11ccc9" exitCode=0
Oct 30 20:54:08.162630 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.162552    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"
Oct 30 20:54:08.163356 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.163339    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"
Oct 30 20:54:08.164014 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.163979    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk" event={"ID":"6295120b-8bbc-4502-8429-4f04d7d5c5a4","Type":"ContainerDied","Data":"45b8103cf080d88ad6ba8310fa33035ccf747e9b9195dc503291a1adba11ccc9"}
Oct 30 20:54:08.169905 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.169881    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk" event={"ID":"6295120b-8bbc-4502-8429-4f04d7d5c5a4","Type":"ContainerDied","Data":"fd8c9cdfd3092aed8afff9f5a59bb6acccf010b0750ba2b3a9d8e5e0071a1be6"}
Oct 30 20:54:08.170084 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.170052    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42" event={"ID":"c4d634d9-39de-4614-9599-38adf86829e2","Type":"ContainerDied","Data":"ccecb5607c37595ea3fe07271e177ea3db15296bbeecbed3def11b721934dda7"}
Oct 30 20:54:08.170191 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.169588    5071 generic.go:356] "Generic (PLEG): container finished" podID="c4d634d9-39de-4614-9599-38adf86829e2" containerID="ccecb5607c37595ea3fe07271e177ea3db15296bbeecbed3def11b721934dda7" exitCode=0
Oct 30 20:54:08.170434 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.164248    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"
Oct 30 20:54:08.170610 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.170598    5071 scope.go:122] "RemoveContainer" containerID="45b8103cf080d88ad6ba8310fa33035ccf747e9b9195dc503291a1adba11ccc9"
Oct 30 20:54:08.173214 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.172895    5071 generic.go:356] "Generic (PLEG): container finished" podID="191943aa-3324-4132-9edf-d2fbfc748b87" containerID="98f0c3af6a8bcfce29ea2314e71d8449f351240bbbb1d90cad38aeec82338608" exitCode=0
Oct 30 20:54:08.173214 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.173068    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8" event={"ID":"191943aa-3324-4132-9edf-d2fbfc748b87","Type":"ContainerDied","Data":"98f0c3af6a8bcfce29ea2314e71d8449f351240bbbb1d90cad38aeec82338608"}
Oct 30 20:54:08.179354 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.179322    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6295120b-8bbc-4502-8429-4f04d7d5c5a4-istio-envoy" pod "6295120b-8bbc-4502-8429-4f04d7d5c5a4" (UID: "6295120b-8bbc-4502-8429-4f04d7d5c5a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:08.182046 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.181985    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6295120b-8bbc-4502-8429-4f04d7d5c5a4-istio-podinfo" pod "6295120b-8bbc-4502-8429-4f04d7d5c5a4" (UID: "6295120b-8bbc-4502-8429-4f04d7d5c5a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:54:08.201472 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.201440    5071 scope.go:122] "RemoveContainer" containerID="45b8103cf080d88ad6ba8310fa33035ccf747e9b9195dc503291a1adba11ccc9"
Oct 30 20:54:08.203159 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:54:08.203128    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45b8103cf080d88ad6ba8310fa33035ccf747e9b9195dc503291a1adba11ccc9\": container with ID starting with 45b8103cf080d88ad6ba8310fa33035ccf747e9b9195dc503291a1adba11ccc9 not found: ID does not exist" containerID="45b8103cf080d88ad6ba8310fa33035ccf747e9b9195dc503291a1adba11ccc9"
Oct 30 20:54:08.203269 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.203166    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45b8103cf080d88ad6ba8310fa33035ccf747e9b9195dc503291a1adba11ccc9"} err="failed to get container status \"45b8103cf080d88ad6ba8310fa33035ccf747e9b9195dc503291a1adba11ccc9\": rpc error: code = NotFound desc = could not find container \"45b8103cf080d88ad6ba8310fa33035ccf747e9b9195dc503291a1adba11ccc9\": container with ID starting with 45b8103cf080d88ad6ba8310fa33035ccf747e9b9195dc503291a1adba11ccc9 not found: ID does not exist"
Oct 30 20:54:08.203269 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.203184    5071 scope.go:122] "RemoveContainer" containerID="ccecb5607c37595ea3fe07271e177ea3db15296bbeecbed3def11b721934dda7"
Oct 30 20:54:08.209157 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.209136    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c82rd\" (UniqueName: \"kubernetes.io/projected/6295120b-8bbc-4502-8429-4f04d7d5c5a4-kube-api-access-c82rd\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:08.209267 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.209256    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6295120b-8bbc-4502-8429-4f04d7d5c5a4-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:08.209331 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.209320    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6295120b-8bbc-4502-8429-4f04d7d5c5a4-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:08.217653 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.217630    5071 scope.go:122] "RemoveContainer" containerID="98f0c3af6a8bcfce29ea2314e71d8449f351240bbbb1d90cad38aeec82338608"
Oct 30 20:54:08.309898 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.309862    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/191943aa-3324-4132-9edf-d2fbfc748b87-kube-api-access-7gcvf\" (UniqueName: \"kubernetes.io/projected/191943aa-3324-4132-9edf-d2fbfc748b87-kube-api-access-7gcvf\") pod \"191943aa-3324-4132-9edf-d2fbfc748b87\" (UID: \"191943aa-3324-4132-9edf-d2fbfc748b87\") "
Oct 30 20:54:08.310070 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.309934    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4d634d9-39de-4614-9599-38adf86829e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4d634d9-39de-4614-9599-38adf86829e2-istio-envoy\") pod \"c4d634d9-39de-4614-9599-38adf86829e2\" (UID: \"c4d634d9-39de-4614-9599-38adf86829e2\") "
Oct 30 20:54:08.310070 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.309981    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/191943aa-3324-4132-9edf-d2fbfc748b87-istio-token\" (UniqueName: \"kubernetes.io/projected/191943aa-3324-4132-9edf-d2fbfc748b87-istio-token\") pod \"191943aa-3324-4132-9edf-d2fbfc748b87\" (UID: \"191943aa-3324-4132-9edf-d2fbfc748b87\") "
Oct 30 20:54:08.310070 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.310057    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4d634d9-39de-4614-9599-38adf86829e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4d634d9-39de-4614-9599-38adf86829e2-istiod-ca-cert\") pod \"c4d634d9-39de-4614-9599-38adf86829e2\" (UID: \"c4d634d9-39de-4614-9599-38adf86829e2\") "
Oct 30 20:54:08.310252 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.310118    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/191943aa-3324-4132-9edf-d2fbfc748b87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/191943aa-3324-4132-9edf-d2fbfc748b87-istio-data\") pod \"191943aa-3324-4132-9edf-d2fbfc748b87\" (UID: \"191943aa-3324-4132-9edf-d2fbfc748b87\") "
Oct 30 20:54:08.310252 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.310149    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4d634d9-39de-4614-9599-38adf86829e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d634d9-39de-4614-9599-38adf86829e2-credential-socket\") pod \"c4d634d9-39de-4614-9599-38adf86829e2\" (UID: \"c4d634d9-39de-4614-9599-38adf86829e2\") "
Oct 30 20:54:08.310252 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.310189    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/191943aa-3324-4132-9edf-d2fbfc748b87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/191943aa-3324-4132-9edf-d2fbfc748b87-istiod-ca-cert\") pod \"191943aa-3324-4132-9edf-d2fbfc748b87\" (UID: \"191943aa-3324-4132-9edf-d2fbfc748b87\") "
Oct 30 20:54:08.310252 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.310240    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/191943aa-3324-4132-9edf-d2fbfc748b87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/191943aa-3324-4132-9edf-d2fbfc748b87-credential-socket\") pod \"191943aa-3324-4132-9edf-d2fbfc748b87\" (UID: \"191943aa-3324-4132-9edf-d2fbfc748b87\") "
Oct 30 20:54:08.311006 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.310269    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/191943aa-3324-4132-9edf-d2fbfc748b87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/191943aa-3324-4132-9edf-d2fbfc748b87-istio-podinfo\") pod \"191943aa-3324-4132-9edf-d2fbfc748b87\" (UID: \"191943aa-3324-4132-9edf-d2fbfc748b87\") "
Oct 30 20:54:08.311006 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.310306    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4d634d9-39de-4614-9599-38adf86829e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d634d9-39de-4614-9599-38adf86829e2-workload-socket\") pod \"c4d634d9-39de-4614-9599-38adf86829e2\" (UID: \"c4d634d9-39de-4614-9599-38adf86829e2\") "
Oct 30 20:54:08.311006 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.310370    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4d634d9-39de-4614-9599-38adf86829e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4d634d9-39de-4614-9599-38adf86829e2-istio-podinfo\") pod \"c4d634d9-39de-4614-9599-38adf86829e2\" (UID: \"c4d634d9-39de-4614-9599-38adf86829e2\") "
Oct 30 20:54:08.311006 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.310495    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4d634d9-39de-4614-9599-38adf86829e2-kube-api-access-x2hcg\" (UniqueName: \"kubernetes.io/projected/c4d634d9-39de-4614-9599-38adf86829e2-kube-api-access-x2hcg\") pod \"c4d634d9-39de-4614-9599-38adf86829e2\" (UID: \"c4d634d9-39de-4614-9599-38adf86829e2\") "
Oct 30 20:54:08.311006 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.310554    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/191943aa-3324-4132-9edf-d2fbfc748b87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/191943aa-3324-4132-9edf-d2fbfc748b87-workload-socket\") pod \"191943aa-3324-4132-9edf-d2fbfc748b87\" (UID: \"191943aa-3324-4132-9edf-d2fbfc748b87\") "
Oct 30 20:54:08.311006 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.310601    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/191943aa-3324-4132-9edf-d2fbfc748b87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/191943aa-3324-4132-9edf-d2fbfc748b87-workload-certs\") pod \"191943aa-3324-4132-9edf-d2fbfc748b87\" (UID: \"191943aa-3324-4132-9edf-d2fbfc748b87\") "
Oct 30 20:54:08.311006 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.310674    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4d634d9-39de-4614-9599-38adf86829e2-istio-token\" (UniqueName: \"kubernetes.io/projected/c4d634d9-39de-4614-9599-38adf86829e2-istio-token\") pod \"c4d634d9-39de-4614-9599-38adf86829e2\" (UID: \"c4d634d9-39de-4614-9599-38adf86829e2\") "
Oct 30 20:54:08.311006 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.310718    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/191943aa-3324-4132-9edf-d2fbfc748b87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/191943aa-3324-4132-9edf-d2fbfc748b87-istio-envoy\") pod \"191943aa-3324-4132-9edf-d2fbfc748b87\" (UID: \"191943aa-3324-4132-9edf-d2fbfc748b87\") "
Oct 30 20:54:08.311006 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.310745    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4d634d9-39de-4614-9599-38adf86829e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4d634d9-39de-4614-9599-38adf86829e2-istio-data\") pod \"c4d634d9-39de-4614-9599-38adf86829e2\" (UID: \"c4d634d9-39de-4614-9599-38adf86829e2\") "
Oct 30 20:54:08.311006 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.310755    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/191943aa-3324-4132-9edf-d2fbfc748b87-istiod-ca-cert" pod "191943aa-3324-4132-9edf-d2fbfc748b87" (UID: "191943aa-3324-4132-9edf-d2fbfc748b87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:08.311006 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.310791    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4d634d9-39de-4614-9599-38adf86829e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4d634d9-39de-4614-9599-38adf86829e2-workload-certs\") pod \"c4d634d9-39de-4614-9599-38adf86829e2\" (UID: \"c4d634d9-39de-4614-9599-38adf86829e2\") "
Oct 30 20:54:08.311588 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.311181    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4d634d9-39de-4614-9599-38adf86829e2-istiod-ca-cert" pod "c4d634d9-39de-4614-9599-38adf86829e2" (UID: "c4d634d9-39de-4614-9599-38adf86829e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:08.311588 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.311483    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191943aa-3324-4132-9edf-d2fbfc748b87-istio-data" pod "191943aa-3324-4132-9edf-d2fbfc748b87" (UID: "191943aa-3324-4132-9edf-d2fbfc748b87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:08.311890 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.311870    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191943aa-3324-4132-9edf-d2fbfc748b87-credential-socket" pod "191943aa-3324-4132-9edf-d2fbfc748b87" (UID: "191943aa-3324-4132-9edf-d2fbfc748b87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:08.312022 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.312000    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4d634d9-39de-4614-9599-38adf86829e2-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:08.312103 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.312029    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/191943aa-3324-4132-9edf-d2fbfc748b87-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:08.312103 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.312045    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/191943aa-3324-4132-9edf-d2fbfc748b87-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:08.313296 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.312245    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4d634d9-39de-4614-9599-38adf86829e2-credential-socket" pod "c4d634d9-39de-4614-9599-38adf86829e2" (UID: "c4d634d9-39de-4614-9599-38adf86829e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:08.313296 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.313046    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4d634d9-39de-4614-9599-38adf86829e2-workload-certs" pod "c4d634d9-39de-4614-9599-38adf86829e2" (UID: "c4d634d9-39de-4614-9599-38adf86829e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:08.315626 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.313810    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/191943aa-3324-4132-9edf-d2fbfc748b87-kube-api-access-7gcvf" pod "191943aa-3324-4132-9edf-d2fbfc748b87" (UID: "191943aa-3324-4132-9edf-d2fbfc748b87"). InnerVolumeSpecName "kube-api-access-7gcvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:08.315626 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.314556    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/191943aa-3324-4132-9edf-d2fbfc748b87-istio-podinfo" pod "191943aa-3324-4132-9edf-d2fbfc748b87" (UID: "191943aa-3324-4132-9edf-d2fbfc748b87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:54:08.315626 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.314726    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4d634d9-39de-4614-9599-38adf86829e2-istio-data" pod "c4d634d9-39de-4614-9599-38adf86829e2" (UID: "c4d634d9-39de-4614-9599-38adf86829e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:08.315626 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.314933    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4d634d9-39de-4614-9599-38adf86829e2-workload-socket" pod "c4d634d9-39de-4614-9599-38adf86829e2" (UID: "c4d634d9-39de-4614-9599-38adf86829e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:08.315626 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.315107    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191943aa-3324-4132-9edf-d2fbfc748b87-workload-socket" pod "191943aa-3324-4132-9edf-d2fbfc748b87" (UID: "191943aa-3324-4132-9edf-d2fbfc748b87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:08.315626 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.315204    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/191943aa-3324-4132-9edf-d2fbfc748b87-istio-token" pod "191943aa-3324-4132-9edf-d2fbfc748b87" (UID: "191943aa-3324-4132-9edf-d2fbfc748b87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:08.315626 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.315315    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191943aa-3324-4132-9edf-d2fbfc748b87-workload-certs" pod "191943aa-3324-4132-9edf-d2fbfc748b87" (UID: "191943aa-3324-4132-9edf-d2fbfc748b87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:08.315626 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.315534    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4d634d9-39de-4614-9599-38adf86829e2-istio-token" pod "c4d634d9-39de-4614-9599-38adf86829e2" (UID: "c4d634d9-39de-4614-9599-38adf86829e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:08.315626 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.315597    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191943aa-3324-4132-9edf-d2fbfc748b87-istio-envoy" pod "191943aa-3324-4132-9edf-d2fbfc748b87" (UID: "191943aa-3324-4132-9edf-d2fbfc748b87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:08.316632 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.316609    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4d634d9-39de-4614-9599-38adf86829e2-kube-api-access-x2hcg" pod "c4d634d9-39de-4614-9599-38adf86829e2" (UID: "c4d634d9-39de-4614-9599-38adf86829e2"). InnerVolumeSpecName "kube-api-access-x2hcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:08.317140 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.317113    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4d634d9-39de-4614-9599-38adf86829e2-istio-podinfo" pod "c4d634d9-39de-4614-9599-38adf86829e2" (UID: "c4d634d9-39de-4614-9599-38adf86829e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:54:08.318243 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.318221    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4d634d9-39de-4614-9599-38adf86829e2-istio-envoy" pod "c4d634d9-39de-4614-9599-38adf86829e2" (UID: "c4d634d9-39de-4614-9599-38adf86829e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:08.413959 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.413922    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d634d9-39de-4614-9599-38adf86829e2-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:08.413959 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.413963    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/191943aa-3324-4132-9edf-d2fbfc748b87-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:08.414388 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.413978    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/191943aa-3324-4132-9edf-d2fbfc748b87-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:08.414388 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.413991    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d634d9-39de-4614-9599-38adf86829e2-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:08.414388 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.414003    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4d634d9-39de-4614-9599-38adf86829e2-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:08.414388 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.414016    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2hcg\" (UniqueName: \"kubernetes.io/projected/c4d634d9-39de-4614-9599-38adf86829e2-kube-api-access-x2hcg\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:08.414388 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.414030    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/191943aa-3324-4132-9edf-d2fbfc748b87-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:08.414388 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.414043    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/191943aa-3324-4132-9edf-d2fbfc748b87-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:08.414388 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.414056    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4d634d9-39de-4614-9599-38adf86829e2-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:08.414388 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.414069    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/191943aa-3324-4132-9edf-d2fbfc748b87-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:08.414388 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.414084    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4d634d9-39de-4614-9599-38adf86829e2-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:08.414388 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.414099    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4d634d9-39de-4614-9599-38adf86829e2-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:08.414388 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.414113    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gcvf\" (UniqueName: \"kubernetes.io/projected/191943aa-3324-4132-9edf-d2fbfc748b87-kube-api-access-7gcvf\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:08.414388 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.414130    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4d634d9-39de-4614-9599-38adf86829e2-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:08.414388 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.414142    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/191943aa-3324-4132-9edf-d2fbfc748b87-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:08.628109 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.628077    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"]
Oct 30 20:54:08.632185 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.632158    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-fqngk"]
Oct 30 20:54:08.840953 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:08.840917    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:09.182306 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:09.182274    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"
Oct 30 20:54:09.182473 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:09.182308    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42" event={"ID":"c4d634d9-39de-4614-9599-38adf86829e2","Type":"ContainerDied","Data":"826c0ee5498498410d9545468f8f6913141cfa5644fac804d48ff3493b8e0ee5"}
Oct 30 20:54:09.183422 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:09.183403    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"
Oct 30 20:54:09.183545 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:09.183448    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8" event={"ID":"191943aa-3324-4132-9edf-d2fbfc748b87","Type":"ContainerDied","Data":"11d149d931b39b83d6d7d44857eab0f9f05668ebb790fdf4279702dffba27698"}
Oct 30 20:54:09.351334 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:09.351219    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"]
Oct 30 20:54:09.356974 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:09.356946    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-ql2c8"]
Oct 30 20:54:09.468295 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:09.468215    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"]
Oct 30 20:54:09.472489 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:09.472465    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-5zw42"]
Oct 30 20:54:10.205784 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:10.205743    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="191943aa-3324-4132-9edf-d2fbfc748b87" path="/var/lib/kubelet/pods/191943aa-3324-4132-9edf-d2fbfc748b87/volumes"
Oct 30 20:54:10.206626 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:10.206600    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6295120b-8bbc-4502-8429-4f04d7d5c5a4" path="/var/lib/kubelet/pods/6295120b-8bbc-4502-8429-4f04d7d5c5a4/volumes"
Oct 30 20:54:10.207446 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:10.207424    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4d634d9-39de-4614-9599-38adf86829e2" path="/var/lib/kubelet/pods/c4d634d9-39de-4614-9599-38adf86829e2/volumes"
Oct 30 20:54:12.888271 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:12.888241    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3d7c11-5d6df9947f-xrgfn"
Oct 30 20:54:12.888656 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:12.888317    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3d7c11-698dd87b66-z282h"
Oct 30 20:54:13.638697 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:13.638662    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-b32941-66c49c7d84-pj6jk" podUID="6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:54:13.655676 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:13.655642    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b32941-57bff4bc8c-qgdg8" podUID="f218d322-bfbb-4396-8152-5b1138f476e8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:54:14.262618 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:14.262585    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:14.511615 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:14.511577    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:14.792557 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:14.792509    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:15.385612 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.385550    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"]
Oct 30 20:54:15.402241 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.402207    5071 cpu_manager.go:414] "containerMap: removing container" podUID="6295120b-8bbc-4502-8429-4f04d7d5c5a4" containerName="istio-proxy"
Oct 30 20:54:15.402433 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.402415    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6295120b-8bbc-4502-8429-4f04d7d5c5a4" containerName="istio-proxy"
Oct 30 20:54:15.402548 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.402536    5071 cpu_manager.go:414] "containerMap: removing container" podUID="191943aa-3324-4132-9edf-d2fbfc748b87" containerName="istio-proxy"
Oct 30 20:54:15.402659 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.402646    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="191943aa-3324-4132-9edf-d2fbfc748b87" containerName="istio-proxy"
Oct 30 20:54:15.405915 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.405886    5071 cpu_manager.go:414] "containerMap: removing container" podUID="07b0cd67-3dfd-47c0-b4d5-df6c3d61946b" containerName="payload-pre-processing"
Oct 30 20:54:15.406071 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.406056    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07b0cd67-3dfd-47c0-b4d5-df6c3d61946b" containerName="payload-pre-processing"
Oct 30 20:54:15.406173 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.406162    5071 cpu_manager.go:414] "containerMap: removing container" podUID="c4d634d9-39de-4614-9599-38adf86829e2" containerName="istio-proxy"
Oct 30 20:54:15.406250 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.406239    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4d634d9-39de-4614-9599-38adf86829e2" containerName="istio-proxy"
Oct 30 20:54:15.406329 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.406319    5071 cpu_manager.go:414] "containerMap: removing container" podUID="7947ba4c-bbde-45ff-a524-e00af39df6e6" containerName="payload-processing"
Oct 30 20:54:15.406422 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.406411    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7947ba4c-bbde-45ff-a524-e00af39df6e6" containerName="payload-processing"
Oct 30 20:54:15.417890 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.406715    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="6295120b-8bbc-4502-8429-4f04d7d5c5a4" containerName="istio-proxy"
Oct 30 20:54:15.418135 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.418114    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="191943aa-3324-4132-9edf-d2fbfc748b87" containerName="istio-proxy"
Oct 30 20:54:15.418258 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.418247    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="07b0cd67-3dfd-47c0-b4d5-df6c3d61946b" containerName="payload-pre-processing"
Oct 30 20:54:15.418344 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.418334    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4d634d9-39de-4614-9599-38adf86829e2" containerName="istio-proxy"
Oct 30 20:54:15.418423 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.418413    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="7947ba4c-bbde-45ff-a524-e00af39df6e6" containerName="payload-processing"
Oct 30 20:54:15.438210 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.438180    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:54:15.473997 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.466687    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:15.624903 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.619762    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"]
Oct 30 20:54:15.624903 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.622383    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79131253-b9a6-41dc-b1d4-aa525c4c5e04-istio-envoy\") pod \"e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4\" (UID: \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\") " pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:54:15.624903 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.622446    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79131253-b9a6-41dc-b1d4-aa525c4c5e04-istiod-ca-cert\") pod \"e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4\" (UID: \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\") " pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:54:15.624903 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.622550    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79131253-b9a6-41dc-b1d4-aa525c4c5e04-workload-socket\") pod \"e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4\" (UID: \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\") " pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:54:15.624903 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.622585    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79131253-b9a6-41dc-b1d4-aa525c4c5e04-credential-socket\") pod \"e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4\" (UID: \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\") " pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:54:15.624903 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.622679    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79131253-b9a6-41dc-b1d4-aa525c4c5e04-workload-certs\") pod \"e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4\" (UID: \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\") " pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:54:15.624903 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.622753    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79131253-b9a6-41dc-b1d4-aa525c4c5e04-istio-token\") pod \"e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4\" (UID: \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\") " pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:54:15.624903 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.622798    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79131253-b9a6-41dc-b1d4-aa525c4c5e04-istio-data\") pod \"e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4\" (UID: \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\") " pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:54:15.624903 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.622845    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79131253-b9a6-41dc-b1d4-aa525c4c5e04-istio-podinfo\") pod \"e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4\" (UID: \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\") " pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:54:15.624903 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.622921    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd582\" (UniqueName: \"kubernetes.io/projected/79131253-b9a6-41dc-b1d4-aa525c4c5e04-kube-api-access-rd582\") pod \"e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4\" (UID: \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\") " pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:54:15.782625 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.641619    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-15199a-openshift-default-dockercfg-cdrrv\""
Oct 30 20:54:15.948462 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.931600    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79131253-b9a6-41dc-b1d4-aa525c4c5e04-istio-token\") pod \"e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4\" (UID: \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\") " pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:54:15.948462 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.931690    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79131253-b9a6-41dc-b1d4-aa525c4c5e04-istio-data\") pod \"e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4\" (UID: \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\") " pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:54:15.948462 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.931731    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79131253-b9a6-41dc-b1d4-aa525c4c5e04-istio-podinfo\") pod \"e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4\" (UID: \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\") " pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:54:15.948462 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.931844    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd582\" (UniqueName: \"kubernetes.io/projected/79131253-b9a6-41dc-b1d4-aa525c4c5e04-kube-api-access-rd582\") pod \"e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4\" (UID: \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\") " pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:54:15.948462 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.932013    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79131253-b9a6-41dc-b1d4-aa525c4c5e04-istio-envoy\") pod \"e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4\" (UID: \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\") " pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:54:15.948462 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.932085    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79131253-b9a6-41dc-b1d4-aa525c4c5e04-istiod-ca-cert\") pod \"e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4\" (UID: \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\") " pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:54:15.948462 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.932200    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79131253-b9a6-41dc-b1d4-aa525c4c5e04-workload-socket\") pod \"e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4\" (UID: \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\") " pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:54:15.948462 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.932245    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79131253-b9a6-41dc-b1d4-aa525c4c5e04-credential-socket\") pod \"e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4\" (UID: \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\") " pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:54:15.948462 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.932360    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79131253-b9a6-41dc-b1d4-aa525c4c5e04-workload-certs\") pod \"e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4\" (UID: \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\") " pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:54:15.948462 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.935841    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79131253-b9a6-41dc-b1d4-aa525c4c5e04-istio-data\") pod \"e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4\" (UID: \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\") " pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:54:15.948462 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.943447    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79131253-b9a6-41dc-b1d4-aa525c4c5e04-istiod-ca-cert\") pod \"e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4\" (UID: \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\") " pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:54:15.956102 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.956068    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79131253-b9a6-41dc-b1d4-aa525c4c5e04-workload-socket\") pod \"e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4\" (UID: \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\") " pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:54:15.956585 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.956562    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79131253-b9a6-41dc-b1d4-aa525c4c5e04-credential-socket\") pod \"e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4\" (UID: \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\") " pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:54:15.971449 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.971421    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79131253-b9a6-41dc-b1d4-aa525c4c5e04-istio-envoy\") pod \"e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4\" (UID: \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\") " pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:54:15.975861 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.975808    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79131253-b9a6-41dc-b1d4-aa525c4c5e04-workload-certs\") pod \"e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4\" (UID: \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\") " pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:54:15.992457 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.992414    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79131253-b9a6-41dc-b1d4-aa525c4c5e04-istio-podinfo\") pod \"e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4\" (UID: \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\") " pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:54:16.000073 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:15.999553    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd582\" (UniqueName: \"kubernetes.io/projected/79131253-b9a6-41dc-b1d4-aa525c4c5e04-kube-api-access-rd582\") pod \"e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4\" (UID: \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\") " pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:54:16.013341 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.013283    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79131253-b9a6-41dc-b1d4-aa525c4c5e04-istio-token\") pod \"e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4\" (UID: \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\") " pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:54:16.065593 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.065563    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b32941-66c49c7d84-pj6jk"
Oct 30 20:54:16.200871 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.194220    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:54:16.207133 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.207084    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b32941-57bff4bc8c-qgdg8"
Oct 30 20:54:16.250036 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.241257    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f-plugins-config-volume\") pod \"6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f\" (UID: \"6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f\") "
Oct 30 20:54:16.250036 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.241470    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f-kube-api-access-lgjhv\" (UniqueName: \"kubernetes.io/projected/6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f-kube-api-access-lgjhv\") pod \"6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f\" (UID: \"6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f\") "
Oct 30 20:54:16.250036 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.244063    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f-plugins-config-volume" pod "6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f" (UID: "6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:16.250036 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.244990    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f-kube-api-access-lgjhv" pod "6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f" (UID: "6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f"). InnerVolumeSpecName "kube-api-access-lgjhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:16.250036 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.246266    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:16.254766 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.254746    5071 generic.go:356] "Generic (PLEG): container finished" podID="6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f" containerID="47aa1564ddf759b7df0a648ac36dc75f1160382b129c0a3eece71243f10f06ca" exitCode=0
Oct 30 20:54:16.254998 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.254976    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b32941-66c49c7d84-pj6jk"
Oct 30 20:54:16.256519 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.256497    5071 generic.go:356] "Generic (PLEG): container finished" podID="f218d322-bfbb-4396-8152-5b1138f476e8" containerID="52f1f49ab17276a0da6c8db5706a859d4b7d7112b3037bff95cb673509f7aa25" exitCode=0
Oct 30 20:54:16.256679 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.256668    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b32941-57bff4bc8c-qgdg8"
Oct 30 20:54:16.279884 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.279856    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b32941-66c49c7d84-pj6jk" event={"ID":"6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f","Type":"ContainerDied","Data":"47aa1564ddf759b7df0a648ac36dc75f1160382b129c0a3eece71243f10f06ca"}
Oct 30 20:54:16.280022 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.279913    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b32941-66c49c7d84-pj6jk" event={"ID":"6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f","Type":"ContainerDied","Data":"6d5b491c242948419fb964331604aea47009441e786d2e0d4acf5fcbd47bc247"}
Oct 30 20:54:16.280022 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.279939    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b32941-57bff4bc8c-qgdg8" event={"ID":"f218d322-bfbb-4396-8152-5b1138f476e8","Type":"ContainerDied","Data":"52f1f49ab17276a0da6c8db5706a859d4b7d7112b3037bff95cb673509f7aa25"}
Oct 30 20:54:16.280022 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.279965    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b32941-57bff4bc8c-qgdg8" event={"ID":"f218d322-bfbb-4396-8152-5b1138f476e8","Type":"ContainerDied","Data":"56fff16e7121d9690657f791f6d25ca64bbf1d73b04ca8b4dd9a2ba557cc388a"}
Oct 30 20:54:16.280022 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.279997    5071 scope.go:122] "RemoveContainer" containerID="47aa1564ddf759b7df0a648ac36dc75f1160382b129c0a3eece71243f10f06ca"
Oct 30 20:54:16.294100 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.294078    5071 scope.go:122] "RemoveContainer" containerID="47aa1564ddf759b7df0a648ac36dc75f1160382b129c0a3eece71243f10f06ca"
Oct 30 20:54:16.294423 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:54:16.294401    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47aa1564ddf759b7df0a648ac36dc75f1160382b129c0a3eece71243f10f06ca\": container with ID starting with 47aa1564ddf759b7df0a648ac36dc75f1160382b129c0a3eece71243f10f06ca not found: ID does not exist" containerID="47aa1564ddf759b7df0a648ac36dc75f1160382b129c0a3eece71243f10f06ca"
Oct 30 20:54:16.294488 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.294432    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47aa1564ddf759b7df0a648ac36dc75f1160382b129c0a3eece71243f10f06ca"} err="failed to get container status \"47aa1564ddf759b7df0a648ac36dc75f1160382b129c0a3eece71243f10f06ca\": rpc error: code = NotFound desc = could not find container \"47aa1564ddf759b7df0a648ac36dc75f1160382b129c0a3eece71243f10f06ca\": container with ID starting with 47aa1564ddf759b7df0a648ac36dc75f1160382b129c0a3eece71243f10f06ca not found: ID does not exist"
Oct 30 20:54:16.294488 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.294450    5071 scope.go:122] "RemoveContainer" containerID="52f1f49ab17276a0da6c8db5706a859d4b7d7112b3037bff95cb673509f7aa25"
Oct 30 20:54:16.315440 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.315291    5071 scope.go:122] "RemoveContainer" containerID="52f1f49ab17276a0da6c8db5706a859d4b7d7112b3037bff95cb673509f7aa25"
Oct 30 20:54:16.316227 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:54:16.316200    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52f1f49ab17276a0da6c8db5706a859d4b7d7112b3037bff95cb673509f7aa25\": container with ID starting with 52f1f49ab17276a0da6c8db5706a859d4b7d7112b3037bff95cb673509f7aa25 not found: ID does not exist" containerID="52f1f49ab17276a0da6c8db5706a859d4b7d7112b3037bff95cb673509f7aa25"
Oct 30 20:54:16.316340 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.316242    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52f1f49ab17276a0da6c8db5706a859d4b7d7112b3037bff95cb673509f7aa25"} err="failed to get container status \"52f1f49ab17276a0da6c8db5706a859d4b7d7112b3037bff95cb673509f7aa25\": rpc error: code = NotFound desc = could not find container \"52f1f49ab17276a0da6c8db5706a859d4b7d7112b3037bff95cb673509f7aa25\": container with ID starting with 52f1f49ab17276a0da6c8db5706a859d4b7d7112b3037bff95cb673509f7aa25 not found: ID does not exist"
Oct 30 20:54:16.343571 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.343472    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f218d322-bfbb-4396-8152-5b1138f476e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f218d322-bfbb-4396-8152-5b1138f476e8-plugins-config-volume\") pod \"f218d322-bfbb-4396-8152-5b1138f476e8\" (UID: \"f218d322-bfbb-4396-8152-5b1138f476e8\") "
Oct 30 20:54:16.345157 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.344105    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f218d322-bfbb-4396-8152-5b1138f476e8-kube-api-access-rdkl9\" (UniqueName: \"kubernetes.io/projected/f218d322-bfbb-4396-8152-5b1138f476e8-kube-api-access-rdkl9\") pod \"f218d322-bfbb-4396-8152-5b1138f476e8\" (UID: \"f218d322-bfbb-4396-8152-5b1138f476e8\") "
Oct 30 20:54:16.345157 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.344881    5071 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f-plugins-config-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:16.345157 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.344905    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgjhv\" (UniqueName: \"kubernetes.io/projected/6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f-kube-api-access-lgjhv\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:16.345157 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.345095    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f218d322-bfbb-4396-8152-5b1138f476e8-plugins-config-volume" pod "f218d322-bfbb-4396-8152-5b1138f476e8" (UID: "f218d322-bfbb-4396-8152-5b1138f476e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:16.348893 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.348161    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f218d322-bfbb-4396-8152-5b1138f476e8-kube-api-access-rdkl9" pod "f218d322-bfbb-4396-8152-5b1138f476e8" (UID: "f218d322-bfbb-4396-8152-5b1138f476e8"). InnerVolumeSpecName "kube-api-access-rdkl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:16.446922 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.446887    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdkl9\" (UniqueName: \"kubernetes.io/projected/f218d322-bfbb-4396-8152-5b1138f476e8-kube-api-access-rdkl9\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:16.446922 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.446916    5071 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f218d322-bfbb-4396-8152-5b1138f476e8-plugins-config-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:16.454974 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.454939    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b32941-66c49c7d84-pj6jk"]
Oct 30 20:54:16.460644 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.460616    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b32941-66c49c7d84-pj6jk"]
Oct 30 20:54:16.684465 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.684432    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b32941-57bff4bc8c-qgdg8"]
Oct 30 20:54:16.686402 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:54:16.686371    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79131253_b9a6_41dc_b1d4_aa525c4c5e04.slice/crio-130c83204ce27a441a8210a4f418e2c7b6ea4afedd994423fe33dd713fdfd83c WatchSource:0}: Error finding container 130c83204ce27a441a8210a4f418e2c7b6ea4afedd994423fe33dd713fdfd83c: Status 404 returned error can't find the container with id 130c83204ce27a441a8210a4f418e2c7b6ea4afedd994423fe33dd713fdfd83c
Oct 30 20:54:16.689346 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.689326    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"]
Oct 30 20:54:16.690040 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.689999    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:54:16.690162 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.690085    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:54:16.690162 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.690130    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:54:16.693392 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.693334    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b32941-57bff4bc8c-qgdg8"]
Oct 30 20:54:16.736588 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:54:16.736556    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:54:16.761798 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:16.761765    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:17.270906 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:17.270872    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4" event={"ID":"79131253-b9a6-41dc-b1d4-aa525c4c5e04","Type":"ContainerStarted","Data":"5a6213b9ad5df080a05d78effb7cc46f8bc54cec21558f29168e3cec6a358d83"}
Oct 30 20:54:17.271081 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:17.270924    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4" event={"ID":"79131253-b9a6-41dc-b1d4-aa525c4c5e04","Type":"ContainerStarted","Data":"130c83204ce27a441a8210a4f418e2c7b6ea4afedd994423fe33dd713fdfd83c"}
Oct 30 20:54:17.408242 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:17.408185    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4" podStartSLOduration=2.408166095 podStartE2EDuration="2.408166095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:17.400334115 +0000 UTC m=+1669.317355650" watchObservedRunningTime="2026-10-30 20:54:17.408166095 +0000 UTC m=+1669.325187630"
Oct 30 20:54:18.208812 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:18.208776    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f" path="/var/lib/kubelet/pods/6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f/volumes"
Oct 30 20:54:18.209454 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:18.209433    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f218d322-bfbb-4396-8152-5b1138f476e8" path="/var/lib/kubelet/pods/f218d322-bfbb-4396-8152-5b1138f476e8/volumes"
Oct 30 20:54:18.209961 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:18.209944    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:54:18.210089 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:18.210069    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:54:18.282057 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:18.282024    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:54:18.283024 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:18.283006    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:54:19.181942 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.181911    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ff20fd9-fg2x8"]
Oct 30 20:54:19.182698 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.182684    5071 cpu_manager.go:414] "containerMap: removing container" podUID="f218d322-bfbb-4396-8152-5b1138f476e8" containerName="payload-pre-processing"
Oct 30 20:54:19.182741 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.182700    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f218d322-bfbb-4396-8152-5b1138f476e8" containerName="payload-pre-processing"
Oct 30 20:54:19.182774 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.182743    5071 cpu_manager.go:414] "containerMap: removing container" podUID="6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f" containerName="payload-processing"
Oct 30 20:54:19.182774 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.182748    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f" containerName="payload-processing"
Oct 30 20:54:19.182894 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.182885    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="6908b42f-ec2a-4943-8a1f-fd8b0ef07e8f" containerName="payload-processing"
Oct 30 20:54:19.182945 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.182901    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="f218d322-bfbb-4396-8152-5b1138f476e8" containerName="payload-pre-processing"
Oct 30 20:54:19.186046 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.186023    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff20fd9-fg2x8"
Oct 30 20:54:19.239682 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.239651    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:54:19.289032 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.289004    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 20:54:19.293928 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.293897    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ff20fd9-fg2x8"]
Oct 30 20:54:19.297290 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.297224    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04143ede-8739-40a6-9b15-45692bc79b6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ff20fd9-fg2x8\" (UID: \"04143ede-8739-40a6-9b15-45692bc79b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff20fd9-fg2x8"
Oct 30 20:54:19.297584 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.297556    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04143ede-8739-40a6-9b15-45692bc79b6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ff20fd9-fg2x8\" (UID: \"04143ede-8739-40a6-9b15-45692bc79b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff20fd9-fg2x8"
Oct 30 20:54:19.297712 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.297691    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvgcf\" (UniqueName: \"kubernetes.io/projected/04143ede-8739-40a6-9b15-45692bc79b6a-kube-api-access-vvgcf\") pod \"maas-api-revoke-sub-3ff20fd9-fg2x8\" (UID: \"04143ede-8739-40a6-9b15-45692bc79b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff20fd9-fg2x8"
Oct 30 20:54:19.399508 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.399475    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04143ede-8739-40a6-9b15-45692bc79b6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ff20fd9-fg2x8\" (UID: \"04143ede-8739-40a6-9b15-45692bc79b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff20fd9-fg2x8"
Oct 30 20:54:19.399702 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.399651    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04143ede-8739-40a6-9b15-45692bc79b6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ff20fd9-fg2x8\" (UID: \"04143ede-8739-40a6-9b15-45692bc79b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff20fd9-fg2x8"
Oct 30 20:54:19.399797 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.399780    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvgcf\" (UniqueName: \"kubernetes.io/projected/04143ede-8739-40a6-9b15-45692bc79b6a-kube-api-access-vvgcf\") pod \"maas-api-revoke-sub-3ff20fd9-fg2x8\" (UID: \"04143ede-8739-40a6-9b15-45692bc79b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff20fd9-fg2x8"
Oct 30 20:54:19.400794 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.400746    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04143ede-8739-40a6-9b15-45692bc79b6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ff20fd9-fg2x8\" (UID: \"04143ede-8739-40a6-9b15-45692bc79b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff20fd9-fg2x8"
Oct 30 20:54:19.420809 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.420776    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04143ede-8739-40a6-9b15-45692bc79b6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ff20fd9-fg2x8\" (UID: \"04143ede-8739-40a6-9b15-45692bc79b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff20fd9-fg2x8"
Oct 30 20:54:19.473561 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.473487    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvgcf\" (UniqueName: \"kubernetes.io/projected/04143ede-8739-40a6-9b15-45692bc79b6a-kube-api-access-vvgcf\") pod \"maas-api-revoke-sub-3ff20fd9-fg2x8\" (UID: \"04143ede-8739-40a6-9b15-45692bc79b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff20fd9-fg2x8"
Oct 30 20:54:19.501240 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.501208    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:19.501983 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.501958    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff20fd9-fg2x8"
Oct 30 20:54:19.670089 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.670060    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r"]
Oct 30 20:54:19.732504 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.711995    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:19.766027 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.765994    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r"]
Oct 30 20:54:19.766202 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.766124    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r"
Oct 30 20:54:19.826477 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.826449    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-0c83a089-serving-cert\""
Oct 30 20:54:19.868197 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.868166    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f53bc10d-bffe-418e-a223-15e324138b78-maas-api-tls\") pod \"maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r\" (UID: \"f53bc10d-bffe-418e-a223-15e324138b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r"
Oct 30 20:54:19.868419 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.868400    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jhnb\" (UniqueName: \"kubernetes.io/projected/f53bc10d-bffe-418e-a223-15e324138b78-kube-api-access-9jhnb\") pod \"maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r\" (UID: \"f53bc10d-bffe-418e-a223-15e324138b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r"
Oct 30 20:54:19.868491 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.868443    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f53bc10d-bffe-418e-a223-15e324138b78-openshift-service-ca\") pod \"maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r\" (UID: \"f53bc10d-bffe-418e-a223-15e324138b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r"
Oct 30 20:54:19.868773 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.868739    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f53bc10d-bffe-418e-a223-15e324138b78-metrics-certs\") pod \"maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r\" (UID: \"f53bc10d-bffe-418e-a223-15e324138b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r"
Oct 30 20:54:19.962511 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.962476    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:19.970151 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.970103    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jhnb\" (UniqueName: \"kubernetes.io/projected/f53bc10d-bffe-418e-a223-15e324138b78-kube-api-access-9jhnb\") pod \"maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r\" (UID: \"f53bc10d-bffe-418e-a223-15e324138b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r"
Oct 30 20:54:19.970295 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.970170    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f53bc10d-bffe-418e-a223-15e324138b78-openshift-service-ca\") pod \"maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r\" (UID: \"f53bc10d-bffe-418e-a223-15e324138b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r"
Oct 30 20:54:19.970381 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.970337    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f53bc10d-bffe-418e-a223-15e324138b78-metrics-certs\") pod \"maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r\" (UID: \"f53bc10d-bffe-418e-a223-15e324138b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r"
Oct 30 20:54:19.970381 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.970369    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f53bc10d-bffe-418e-a223-15e324138b78-maas-api-tls\") pod \"maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r\" (UID: \"f53bc10d-bffe-418e-a223-15e324138b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r"
Oct 30 20:54:19.970992 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.970959    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f53bc10d-bffe-418e-a223-15e324138b78-openshift-service-ca\") pod \"maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r\" (UID: \"f53bc10d-bffe-418e-a223-15e324138b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r"
Oct 30 20:54:19.973055 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.973035    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f53bc10d-bffe-418e-a223-15e324138b78-maas-api-tls\") pod \"maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r\" (UID: \"f53bc10d-bffe-418e-a223-15e324138b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r"
Oct 30 20:54:19.973182 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:19.973162    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f53bc10d-bffe-418e-a223-15e324138b78-metrics-certs\") pod \"maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r\" (UID: \"f53bc10d-bffe-418e-a223-15e324138b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r"
Oct 30 20:54:20.006523 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:54:20.006495    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04143ede_8739_40a6_9b15_45692bc79b6a.slice/crio-a629399bacf7e9efdeb5218ee3bc3b3ea25ee6c4d4984ca07bdcc914be4ef6e0 WatchSource:0}: Error finding container a629399bacf7e9efdeb5218ee3bc3b3ea25ee6c4d4984ca07bdcc914be4ef6e0: Status 404 returned error can't find the container with id a629399bacf7e9efdeb5218ee3bc3b3ea25ee6c4d4984ca07bdcc914be4ef6e0
Oct 30 20:54:20.007066 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:20.007044    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ff20fd9-fg2x8"]
Oct 30 20:54:20.028644 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:20.028618    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jhnb\" (UniqueName: \"kubernetes.io/projected/f53bc10d-bffe-418e-a223-15e324138b78-kube-api-access-9jhnb\") pod \"maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r\" (UID: \"f53bc10d-bffe-418e-a223-15e324138b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r"
Oct 30 20:54:20.091852 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:20.089375    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r"
Oct 30 20:54:20.142874 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:20.142619    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:20.358513 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:20.358473    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:20.493082 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:20.492137    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff20fd9-fg2x8" event={"ID":"04143ede-8739-40a6-9b15-45692bc79b6a","Type":"ContainerStarted","Data":"a629399bacf7e9efdeb5218ee3bc3b3ea25ee6c4d4984ca07bdcc914be4ef6e0"}
Oct 30 20:54:20.548538 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:20.547997    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:20.811415 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:20.713273    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"]
Oct 30 20:54:20.811415 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:20.777258    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:20.811415 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:20.797286    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 20:54:20.979222 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:20.896661    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c36da32-7ded-4d53-8579-f62445fc61a5-istio-envoy\") pod \"e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw\" (UID: \"2c36da32-7ded-4d53-8579-f62445fc61a5\") " pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 20:54:20.979222 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:20.896720    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c36da32-7ded-4d53-8579-f62445fc61a5-workload-certs\") pod \"e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw\" (UID: \"2c36da32-7ded-4d53-8579-f62445fc61a5\") " pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 20:54:20.979222 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:20.896757    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c36da32-7ded-4d53-8579-f62445fc61a5-istio-data\") pod \"e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw\" (UID: \"2c36da32-7ded-4d53-8579-f62445fc61a5\") " pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 20:54:20.979222 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:20.896796    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6h9l\" (UniqueName: \"kubernetes.io/projected/2c36da32-7ded-4d53-8579-f62445fc61a5-kube-api-access-k6h9l\") pod \"e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw\" (UID: \"2c36da32-7ded-4d53-8579-f62445fc61a5\") " pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 20:54:20.979222 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:20.942241    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-d9b3f6-openshift-default-dockercfg-mvjqq\""
Oct 30 20:54:21.009897 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:20.988887    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"]
Oct 30 20:54:21.009897 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:20.994013    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r"]
Oct 30 20:54:21.041079 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:21.040346    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c36da32-7ded-4d53-8579-f62445fc61a5-credential-socket\") pod \"e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw\" (UID: \"2c36da32-7ded-4d53-8579-f62445fc61a5\") " pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 20:54:21.041079 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:21.040501    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c36da32-7ded-4d53-8579-f62445fc61a5-workload-socket\") pod \"e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw\" (UID: \"2c36da32-7ded-4d53-8579-f62445fc61a5\") " pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 20:54:21.041079 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:21.040548    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c36da32-7ded-4d53-8579-f62445fc61a5-istio-token\") pod \"e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw\" (UID: \"2c36da32-7ded-4d53-8579-f62445fc61a5\") " pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 20:54:21.041079 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:21.040578    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c36da32-7ded-4d53-8579-f62445fc61a5-istiod-ca-cert\") pod \"e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw\" (UID: \"2c36da32-7ded-4d53-8579-f62445fc61a5\") " pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 20:54:21.041079 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:21.040760    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c36da32-7ded-4d53-8579-f62445fc61a5-istio-podinfo\") pod \"e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw\" (UID: \"2c36da32-7ded-4d53-8579-f62445fc61a5\") " pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 20:54:21.153056 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:54:21.135627    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf53bc10d_bffe_418e_a223_15e324138b78.slice/crio-9fb6bb191e76422e533cc3402278b1b8c552fe25ff306b3ef3bf8303c1446e31 WatchSource:0}: Error finding container 9fb6bb191e76422e533cc3402278b1b8c552fe25ff306b3ef3bf8303c1446e31: Status 404 returned error can't find the container with id 9fb6bb191e76422e533cc3402278b1b8c552fe25ff306b3ef3bf8303c1446e31
Oct 30 20:54:21.153056 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:21.146007    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c36da32-7ded-4d53-8579-f62445fc61a5-workload-socket\") pod \"e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw\" (UID: \"2c36da32-7ded-4d53-8579-f62445fc61a5\") " pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 20:54:21.153056 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:21.146063    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c36da32-7ded-4d53-8579-f62445fc61a5-istio-token\") pod \"e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw\" (UID: \"2c36da32-7ded-4d53-8579-f62445fc61a5\") " pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 20:54:21.153056 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:21.146095    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c36da32-7ded-4d53-8579-f62445fc61a5-istiod-ca-cert\") pod \"e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw\" (UID: \"2c36da32-7ded-4d53-8579-f62445fc61a5\") " pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 20:54:21.153056 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:21.146254    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c36da32-7ded-4d53-8579-f62445fc61a5-istio-podinfo\") pod \"e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw\" (UID: \"2c36da32-7ded-4d53-8579-f62445fc61a5\") " pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 20:54:21.153056 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:21.146365    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c36da32-7ded-4d53-8579-f62445fc61a5-istio-envoy\") pod \"e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw\" (UID: \"2c36da32-7ded-4d53-8579-f62445fc61a5\") " pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 20:54:21.153056 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:21.146404    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c36da32-7ded-4d53-8579-f62445fc61a5-workload-certs\") pod \"e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw\" (UID: \"2c36da32-7ded-4d53-8579-f62445fc61a5\") " pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 20:54:21.153056 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:21.146440    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c36da32-7ded-4d53-8579-f62445fc61a5-istio-data\") pod \"e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw\" (UID: \"2c36da32-7ded-4d53-8579-f62445fc61a5\") " pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 20:54:21.153056 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:21.146483    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6h9l\" (UniqueName: \"kubernetes.io/projected/2c36da32-7ded-4d53-8579-f62445fc61a5-kube-api-access-k6h9l\") pod \"e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw\" (UID: \"2c36da32-7ded-4d53-8579-f62445fc61a5\") " pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 20:54:21.153056 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:21.146602    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c36da32-7ded-4d53-8579-f62445fc61a5-credential-socket\") pod \"e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw\" (UID: \"2c36da32-7ded-4d53-8579-f62445fc61a5\") " pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 20:54:21.153056 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:21.147199    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c36da32-7ded-4d53-8579-f62445fc61a5-credential-socket\") pod \"e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw\" (UID: \"2c36da32-7ded-4d53-8579-f62445fc61a5\") " pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 20:54:21.153056 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:21.148149    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c36da32-7ded-4d53-8579-f62445fc61a5-workload-socket\") pod \"e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw\" (UID: \"2c36da32-7ded-4d53-8579-f62445fc61a5\") " pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 20:54:21.157020 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:21.156160    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c36da32-7ded-4d53-8579-f62445fc61a5-istiod-ca-cert\") pod \"e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw\" (UID: \"2c36da32-7ded-4d53-8579-f62445fc61a5\") " pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 20:54:21.160576 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:21.160552    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c36da32-7ded-4d53-8579-f62445fc61a5-workload-certs\") pod \"e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw\" (UID: \"2c36da32-7ded-4d53-8579-f62445fc61a5\") " pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 20:54:21.162614 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:21.162590    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c36da32-7ded-4d53-8579-f62445fc61a5-istio-data\") pod \"e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw\" (UID: \"2c36da32-7ded-4d53-8579-f62445fc61a5\") " pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 20:54:21.194847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:21.167100    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c36da32-7ded-4d53-8579-f62445fc61a5-istio-envoy\") pod \"e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw\" (UID: \"2c36da32-7ded-4d53-8579-f62445fc61a5\") " pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 20:54:21.194847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:21.168641    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c36da32-7ded-4d53-8579-f62445fc61a5-istio-podinfo\") pod \"e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw\" (UID: \"2c36da32-7ded-4d53-8579-f62445fc61a5\") " pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 20:54:21.231848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:21.226813    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c36da32-7ded-4d53-8579-f62445fc61a5-istio-token\") pod \"e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw\" (UID: \"2c36da32-7ded-4d53-8579-f62445fc61a5\") " pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 20:54:21.358146 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:21.320535    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6h9l\" (UniqueName: \"kubernetes.io/projected/2c36da32-7ded-4d53-8579-f62445fc61a5-kube-api-access-k6h9l\") pod \"e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw\" (UID: \"2c36da32-7ded-4d53-8579-f62445fc61a5\") " pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 20:54:21.448457 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:21.448207    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 20:54:21.506339 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:21.506306    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r" event={"ID":"f53bc10d-bffe-418e-a223-15e324138b78","Type":"ContainerStarted","Data":"9fb6bb191e76422e533cc3402278b1b8c552fe25ff306b3ef3bf8303c1446e31"}
Oct 30 20:54:21.508970 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:21.508940    5071 generic.go:356] "Generic (PLEG): container finished" podID="04143ede-8739-40a6-9b15-45692bc79b6a" containerID="2b3c96fb795b1d0bf730ea94433f9ac27af3f92ea1aa253ae2d5515a89283a98" exitCode=0
Oct 30 20:54:21.509319 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:21.509222    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff20fd9-fg2x8" event={"ID":"04143ede-8739-40a6-9b15-45692bc79b6a","Type":"ContainerDied","Data":"2b3c96fb795b1d0bf730ea94433f9ac27af3f92ea1aa253ae2d5515a89283a98"}
Oct 30 20:54:21.762766 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:21.758437    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:21.934545 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:54:21.934510    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c36da32_7ded_4d53_8579_f62445fc61a5.slice/crio-f6375e1cdeec890e8b4375d84d50563eb8589a9fa132c7c92263f6855f961bce WatchSource:0}: Error finding container f6375e1cdeec890e8b4375d84d50563eb8589a9fa132c7c92263f6855f961bce: Status 404 returned error can't find the container with id f6375e1cdeec890e8b4375d84d50563eb8589a9fa132c7c92263f6855f961bce
Oct 30 20:54:21.934670 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:21.934545    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"]
Oct 30 20:54:21.937637 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:21.937598    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:54:21.937701 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:21.937686    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:54:21.937740 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:21.937725    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:54:22.521153 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:22.521097    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw" event={"ID":"2c36da32-7ded-4d53-8579-f62445fc61a5","Type":"ContainerStarted","Data":"7c11413bb0d8e2fdbfdd114869b7450d0a54292162df6de3363659ca73b68163"}
Oct 30 20:54:22.521669 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:22.521167    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw" event={"ID":"2c36da32-7ded-4d53-8579-f62445fc61a5","Type":"ContainerStarted","Data":"f6375e1cdeec890e8b4375d84d50563eb8589a9fa132c7c92263f6855f961bce"}
Oct 30 20:54:22.523586 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:22.523553    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r" event={"ID":"f53bc10d-bffe-418e-a223-15e324138b78","Type":"ContainerStarted","Data":"b2697c0891de36f8c66b74fccc9a058f126acb11a49226443313a971bd6ea9c6"}
Oct 30 20:54:22.706448 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:22.700904    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw" podStartSLOduration=2.700806373 podStartE2EDuration="2.700806373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:22.669118122 +0000 UTC m=+1674.586139658" watchObservedRunningTime="2026-10-30 20:54:22.700806373 +0000 UTC m=+1674.617827913"
Oct 30 20:54:22.797804 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:22.797709    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r" podStartSLOduration=2.739920213 podStartE2EDuration="3.79768641s" totalImagesPullingTime="1.057766197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:22.785606125 +0000 UTC m=+1674.702627669" watchObservedRunningTime="2026-10-30 20:54:22.79768641 +0000 UTC m=+1674.714707953"
Oct 30 20:54:22.928335 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:22.928303    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff20fd9-fg2x8"
Oct 30 20:54:23.076886 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:23.075575    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04143ede-8739-40a6-9b15-45692bc79b6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04143ede-8739-40a6-9b15-45692bc79b6a-maas-api-xks-ca\") pod \"04143ede-8739-40a6-9b15-45692bc79b6a\" (UID: \"04143ede-8739-40a6-9b15-45692bc79b6a\") "
Oct 30 20:54:23.076886 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:23.075737    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04143ede-8739-40a6-9b15-45692bc79b6a-kube-api-access-vvgcf\" (UniqueName: \"kubernetes.io/projected/04143ede-8739-40a6-9b15-45692bc79b6a-kube-api-access-vvgcf\") pod \"04143ede-8739-40a6-9b15-45692bc79b6a\" (UID: \"04143ede-8739-40a6-9b15-45692bc79b6a\") "
Oct 30 20:54:23.076886 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:23.075910    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04143ede-8739-40a6-9b15-45692bc79b6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04143ede-8739-40a6-9b15-45692bc79b6a-maas-api-openshift-ca\") pod \"04143ede-8739-40a6-9b15-45692bc79b6a\" (UID: \"04143ede-8739-40a6-9b15-45692bc79b6a\") "
Oct 30 20:54:23.076886 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:23.076798    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04143ede-8739-40a6-9b15-45692bc79b6a-maas-api-openshift-ca" pod "04143ede-8739-40a6-9b15-45692bc79b6a" (UID: "04143ede-8739-40a6-9b15-45692bc79b6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:23.079201 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:23.079176    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04143ede-8739-40a6-9b15-45692bc79b6a-kube-api-access-vvgcf" pod "04143ede-8739-40a6-9b15-45692bc79b6a" (UID: "04143ede-8739-40a6-9b15-45692bc79b6a"). InnerVolumeSpecName "kube-api-access-vvgcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:23.080132 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:23.080110    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04143ede-8739-40a6-9b15-45692bc79b6a-maas-api-xks-ca" pod "04143ede-8739-40a6-9b15-45692bc79b6a" (UID: "04143ede-8739-40a6-9b15-45692bc79b6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:54:23.201561 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:23.185426    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04143ede-8739-40a6-9b15-45692bc79b6a-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:23.201561 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:23.185462    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvgcf\" (UniqueName: \"kubernetes.io/projected/04143ede-8739-40a6-9b15-45692bc79b6a-kube-api-access-vvgcf\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:23.201561 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:23.185493    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04143ede-8739-40a6-9b15-45692bc79b6a-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:23.246847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:23.234049    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:23.461901 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:23.459897    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 20:54:23.474475 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:23.474443    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 20:54:23.677029 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:23.668434    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff20fd9-fg2x8"
Oct 30 20:54:23.690384 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:23.674895    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ff20fd9-fg2x8" event={"ID":"04143ede-8739-40a6-9b15-45692bc79b6a","Type":"ContainerDied","Data":"a629399bacf7e9efdeb5218ee3bc3b3ea25ee6c4d4984ca07bdcc914be4ef6e0"}
Oct 30 20:54:23.690384 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:23.683331    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a629399bacf7e9efdeb5218ee3bc3b3ea25ee6c4d4984ca07bdcc914be4ef6e0"
Oct 30 20:54:23.690384 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:23.683360    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 20:54:23.690384 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:23.683399    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r"
Oct 30 20:54:23.690384 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:23.683457    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 20:54:23.863085 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:23.863037    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:24.080046 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:24.075140    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:24.256950 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:24.256551    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-15199a-79b8f8b875-m8l5v"]
Oct 30 20:54:24.263891 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:24.259634    5071 cpu_manager.go:414] "containerMap: removing container" podUID="04143ede-8739-40a6-9b15-45692bc79b6a" containerName="revoke-keys"
Oct 30 20:54:24.263891 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:24.259664    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04143ede-8739-40a6-9b15-45692bc79b6a" containerName="revoke-keys"
Oct 30 20:54:24.263891 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:24.259939    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="04143ede-8739-40a6-9b15-45692bc79b6a" containerName="revoke-keys"
Oct 30 20:54:24.268849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:24.268194    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-15199a-79b8f8b875-m8l5v"
Oct 30 20:54:24.315860 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:24.314298    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x22b4\" (UniqueName: \"kubernetes.io/projected/281ee523-c301-4b62-a1ed-8d17e88bc88a-kube-api-access-x22b4\") pod \"payload-pre-processing-e2e-shared-b-w5-15199a-79b8f8b875-m8l5v\" (UID: \"281ee523-c301-4b62-a1ed-8d17e88bc88a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-15199a-79b8f8b875-m8l5v"
Oct 30 20:54:24.315860 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:24.314400    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281ee523-c301-4b62-a1ed-8d17e88bc88a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-15199a-79b8f8b875-m8l5v\" (UID: \"281ee523-c301-4b62-a1ed-8d17e88bc88a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-15199a-79b8f8b875-m8l5v"
Oct 30 20:54:24.346808 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:24.332911    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-15199a\""
Oct 30 20:54:24.346808 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:24.339009    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-15199a-79b8f8b875-m8l5v"]
Oct 30 20:54:24.346808 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:24.339184    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-15199a-dockercfg-hz7zk\""
Oct 30 20:54:24.433845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:24.422136    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x22b4\" (UniqueName: \"kubernetes.io/projected/281ee523-c301-4b62-a1ed-8d17e88bc88a-kube-api-access-x22b4\") pod \"payload-pre-processing-e2e-shared-b-w5-15199a-79b8f8b875-m8l5v\" (UID: \"281ee523-c301-4b62-a1ed-8d17e88bc88a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-15199a-79b8f8b875-m8l5v"
Oct 30 20:54:24.433845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:24.422235    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281ee523-c301-4b62-a1ed-8d17e88bc88a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-15199a-79b8f8b875-m8l5v\" (UID: \"281ee523-c301-4b62-a1ed-8d17e88bc88a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-15199a-79b8f8b875-m8l5v"
Oct 30 20:54:24.433845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:24.423397    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281ee523-c301-4b62-a1ed-8d17e88bc88a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-15199a-79b8f8b875-m8l5v\" (UID: \"281ee523-c301-4b62-a1ed-8d17e88bc88a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-15199a-79b8f8b875-m8l5v"
Oct 30 20:54:24.604845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:24.602251    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-15199a-69b969cbfb-whtbl"]
Oct 30 20:54:24.734001 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:24.733135    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-15199a-69b969cbfb-whtbl"
Oct 30 20:54:24.780339 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:24.775582    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-15199a-69b969cbfb-whtbl"]
Oct 30 20:54:24.790548 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:24.790148    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hffj\" (UniqueName: \"kubernetes.io/projected/9cb154a5-d317-4671-aa04-335b5a9060c7-kube-api-access-6hffj\") pod \"payload-processing-e2e-shared-b-w5-15199a-69b969cbfb-whtbl\" (UID: \"9cb154a5-d317-4671-aa04-335b5a9060c7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-15199a-69b969cbfb-whtbl"
Oct 30 20:54:24.790548 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:24.790333    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cb154a5-d317-4671-aa04-335b5a9060c7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-15199a-69b969cbfb-whtbl\" (UID: \"9cb154a5-d317-4671-aa04-335b5a9060c7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-15199a-69b969cbfb-whtbl"
Oct 30 20:54:24.818598 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:24.818561    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x22b4\" (UniqueName: \"kubernetes.io/projected/281ee523-c301-4b62-a1ed-8d17e88bc88a-kube-api-access-x22b4\") pod \"payload-pre-processing-e2e-shared-b-w5-15199a-79b8f8b875-m8l5v\" (UID: \"281ee523-c301-4b62-a1ed-8d17e88bc88a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-15199a-79b8f8b875-m8l5v"
Oct 30 20:54:24.898899 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:24.894175    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hffj\" (UniqueName: \"kubernetes.io/projected/9cb154a5-d317-4671-aa04-335b5a9060c7-kube-api-access-6hffj\") pod \"payload-processing-e2e-shared-b-w5-15199a-69b969cbfb-whtbl\" (UID: \"9cb154a5-d317-4671-aa04-335b5a9060c7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-15199a-69b969cbfb-whtbl"
Oct 30 20:54:24.898899 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:24.894439    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cb154a5-d317-4671-aa04-335b5a9060c7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-15199a-69b969cbfb-whtbl\" (UID: \"9cb154a5-d317-4671-aa04-335b5a9060c7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-15199a-69b969cbfb-whtbl"
Oct 30 20:54:24.912847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:24.909072    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cb154a5-d317-4671-aa04-335b5a9060c7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-15199a-69b969cbfb-whtbl\" (UID: \"9cb154a5-d317-4671-aa04-335b5a9060c7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-15199a-69b969cbfb-whtbl"
Oct 30 20:54:24.963003 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:24.962963    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hffj\" (UniqueName: \"kubernetes.io/projected/9cb154a5-d317-4671-aa04-335b5a9060c7-kube-api-access-6hffj\") pod \"payload-processing-e2e-shared-b-w5-15199a-69b969cbfb-whtbl\" (UID: \"9cb154a5-d317-4671-aa04-335b5a9060c7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-15199a-69b969cbfb-whtbl"
Oct 30 20:54:25.086215 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:25.086174    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-15199a-79b8f8b875-m8l5v"
Oct 30 20:54:25.164240 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:25.161451    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-15199a-69b969cbfb-whtbl"
Oct 30 20:54:25.494651 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:25.494604    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-15199a-79b8f8b875-m8l5v"]
Oct 30 20:54:25.500669 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:54:25.500622    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod281ee523_c301_4b62_a1ed_8d17e88bc88a.slice/crio-11c70064bc3675ea91e11af11a485e9dd288c414912b6db89da5a3d56860c3ba WatchSource:0}: Error finding container 11c70064bc3675ea91e11af11a485e9dd288c414912b6db89da5a3d56860c3ba: Status 404 returned error can't find the container with id 11c70064bc3675ea91e11af11a485e9dd288c414912b6db89da5a3d56860c3ba
Oct 30 20:54:25.602508 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:54:25.602470    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cb154a5_d317_4671_aa04_335b5a9060c7.slice/crio-30acea1ca8e7308b508c2b5cf10c3132383a754ddcc65e0314b7aa8ace0386cd WatchSource:0}: Error finding container 30acea1ca8e7308b508c2b5cf10c3132383a754ddcc65e0314b7aa8ace0386cd: Status 404 returned error can't find the container with id 30acea1ca8e7308b508c2b5cf10c3132383a754ddcc65e0314b7aa8ace0386cd
Oct 30 20:54:25.604067 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:25.604043    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-15199a-69b969cbfb-whtbl"]
Oct 30 20:54:25.858870 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:25.858811    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-15199a-79b8f8b875-m8l5v" event={"ID":"281ee523-c301-4b62-a1ed-8d17e88bc88a","Type":"ContainerStarted","Data":"48a4c482cbf1b30f19ffe3735022eebb3f8a40a1cd1f13a29485162d3266ab33"}
Oct 30 20:54:25.859349 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:25.858891    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-15199a-79b8f8b875-m8l5v" event={"ID":"281ee523-c301-4b62-a1ed-8d17e88bc88a","Type":"ContainerStarted","Data":"11c70064bc3675ea91e11af11a485e9dd288c414912b6db89da5a3d56860c3ba"}
Oct 30 20:54:25.859349 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:25.858924    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-15199a-79b8f8b875-m8l5v"
Oct 30 20:54:25.861043 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:25.861013    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-15199a-69b969cbfb-whtbl" event={"ID":"9cb154a5-d317-4671-aa04-335b5a9060c7","Type":"ContainerStarted","Data":"1076fb6ee6772a19b74dd9097ca02f5c02f8de41d661bdbab6999490c1800bd1"}
Oct 30 20:54:25.861172 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:25.861066    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-15199a-69b969cbfb-whtbl"
Oct 30 20:54:25.861172 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:25.861084    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-15199a-69b969cbfb-whtbl" event={"ID":"9cb154a5-d317-4671-aa04-335b5a9060c7","Type":"ContainerStarted","Data":"30acea1ca8e7308b508c2b5cf10c3132383a754ddcc65e0314b7aa8ace0386cd"}
Oct 30 20:54:25.979079 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:25.979025    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-15199a-79b8f8b875-m8l5v" podStartSLOduration=1.979009992 podStartE2EDuration="1.979009992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:25.973280957 +0000 UTC m=+1677.890302494" watchObservedRunningTime="2026-10-30 20:54:25.979009992 +0000 UTC m=+1677.896031527"
Oct 30 20:54:26.091926 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:26.091863    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-15199a-69b969cbfb-whtbl" podStartSLOduration=2.091840437 podStartE2EDuration="2.091840437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:26.089041944 +0000 UTC m=+1678.006063480" watchObservedRunningTime="2026-10-30 20:54:26.091840437 +0000 UTC m=+1678.008861969"
Oct 30 20:54:26.100037 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:26.100003    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:26.375975 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:26.375928    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:26.554330 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:26.554295    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:26.729015 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:26.728919    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:29.683092 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:29.683047    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:29.850791 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:29.850763    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r"
Oct 30 20:54:29.930935 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:29.930897    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:31.258710 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:31.258663    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:31.778850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:31.764241    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:31.813842 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:31.789084    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d9b3f6-86cfc8c66c-l48wg"]
Oct 30 20:54:32.073847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:32.070041    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d9b3f6-86cfc8c66c-l48wg"]
Oct 30 20:54:32.073847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:32.070273    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d9b3f6-86cfc8c66c-l48wg"
Oct 30 20:54:32.115129 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:32.114870    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:32.141881 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:32.119930    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/940a58d0-5deb-4bc9-8832-5a17fa300539-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-d9b3f6-86cfc8c66c-l48wg\" (UID: \"940a58d0-5deb-4bc9-8832-5a17fa300539\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d9b3f6-86cfc8c66c-l48wg"
Oct 30 20:54:32.141881 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:32.120139    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn59p\" (UniqueName: \"kubernetes.io/projected/940a58d0-5deb-4bc9-8832-5a17fa300539-kube-api-access-sn59p\") pod \"payload-pre-processing-e2e-shared-a-w4-d9b3f6-86cfc8c66c-l48wg\" (UID: \"940a58d0-5deb-4bc9-8832-5a17fa300539\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d9b3f6-86cfc8c66c-l48wg"
Oct 30 20:54:32.152845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:32.142284    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-d9b3f6-845694f564-8jdjx"]
Oct 30 20:54:32.187044 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:32.159952    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-d9b3f6\""
Oct 30 20:54:32.187044 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:32.160725    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-d9b3f6-dockercfg-j5wxf\""
Oct 30 20:54:32.206852 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:32.188083    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-d9b3f6-845694f564-8jdjx"]
Oct 30 20:54:32.221636 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:32.190628    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d9b3f6-845694f564-8jdjx"
Oct 30 20:54:32.325559 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:32.325480    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:32.342215 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:32.342179    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/940a58d0-5deb-4bc9-8832-5a17fa300539-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-d9b3f6-86cfc8c66c-l48wg\" (UID: \"940a58d0-5deb-4bc9-8832-5a17fa300539\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d9b3f6-86cfc8c66c-l48wg"
Oct 30 20:54:32.342377 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:32.342330    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txj2z\" (UniqueName: \"kubernetes.io/projected/902523f4-4967-4ef2-97be-632d3a0c6383-kube-api-access-txj2z\") pod \"payload-processing-e2e-shared-a-w4-d9b3f6-845694f564-8jdjx\" (UID: \"902523f4-4967-4ef2-97be-632d3a0c6383\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d9b3f6-845694f564-8jdjx"
Oct 30 20:54:32.342441 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:32.342395    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn59p\" (UniqueName: \"kubernetes.io/projected/940a58d0-5deb-4bc9-8832-5a17fa300539-kube-api-access-sn59p\") pod \"payload-pre-processing-e2e-shared-a-w4-d9b3f6-86cfc8c66c-l48wg\" (UID: \"940a58d0-5deb-4bc9-8832-5a17fa300539\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d9b3f6-86cfc8c66c-l48wg"
Oct 30 20:54:32.342692 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:32.342670    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/902523f4-4967-4ef2-97be-632d3a0c6383-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-d9b3f6-845694f564-8jdjx\" (UID: \"902523f4-4967-4ef2-97be-632d3a0c6383\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d9b3f6-845694f564-8jdjx"
Oct 30 20:54:32.343232 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:32.343203    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/940a58d0-5deb-4bc9-8832-5a17fa300539-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-d9b3f6-86cfc8c66c-l48wg\" (UID: \"940a58d0-5deb-4bc9-8832-5a17fa300539\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d9b3f6-86cfc8c66c-l48wg"
Oct 30 20:54:32.401874 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:32.401840    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn59p\" (UniqueName: \"kubernetes.io/projected/940a58d0-5deb-4bc9-8832-5a17fa300539-kube-api-access-sn59p\") pod \"payload-pre-processing-e2e-shared-a-w4-d9b3f6-86cfc8c66c-l48wg\" (UID: \"940a58d0-5deb-4bc9-8832-5a17fa300539\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d9b3f6-86cfc8c66c-l48wg"
Oct 30 20:54:32.444432 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:32.444402    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/902523f4-4967-4ef2-97be-632d3a0c6383-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-d9b3f6-845694f564-8jdjx\" (UID: \"902523f4-4967-4ef2-97be-632d3a0c6383\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d9b3f6-845694f564-8jdjx"
Oct 30 20:54:32.444603 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:32.444550    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txj2z\" (UniqueName: \"kubernetes.io/projected/902523f4-4967-4ef2-97be-632d3a0c6383-kube-api-access-txj2z\") pod \"payload-processing-e2e-shared-a-w4-d9b3f6-845694f564-8jdjx\" (UID: \"902523f4-4967-4ef2-97be-632d3a0c6383\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d9b3f6-845694f564-8jdjx"
Oct 30 20:54:32.445186 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:32.445168    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/902523f4-4967-4ef2-97be-632d3a0c6383-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-d9b3f6-845694f564-8jdjx\" (UID: \"902523f4-4967-4ef2-97be-632d3a0c6383\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d9b3f6-845694f564-8jdjx"
Oct 30 20:54:32.502852 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:32.502804    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txj2z\" (UniqueName: \"kubernetes.io/projected/902523f4-4967-4ef2-97be-632d3a0c6383-kube-api-access-txj2z\") pod \"payload-processing-e2e-shared-a-w4-d9b3f6-845694f564-8jdjx\" (UID: \"902523f4-4967-4ef2-97be-632d3a0c6383\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d9b3f6-845694f564-8jdjx"
Oct 30 20:54:32.597758 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:32.597675    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d9b3f6-86cfc8c66c-l48wg"
Oct 30 20:54:32.619042 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:32.619012    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d9b3f6-845694f564-8jdjx"
Oct 30 20:54:32.867060 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:32.866977    5071 scope.go:122] "RemoveContainer" containerID="09cda05ac8d3877a10953680fa99880359ce54e4c0486e41433684be7ab300b0"
Oct 30 20:54:32.878924 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:32.878903    5071 scope.go:122] "RemoveContainer" containerID="4d82c0b625f6b06529653739fbe13d46973f9437e72b272099b1e795d5943eb3"
Oct 30 20:54:32.889913 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:32.889891    5071 scope.go:122] "RemoveContainer" containerID="31181f4bc6bded46f81703f090860136d71ecfdf46a2c4e94fcf738d02fa7525"
Oct 30 20:54:32.899761 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:32.899739    5071 scope.go:122] "RemoveContainer" containerID="0d6e686b847f788e45f87077c3140994af323c3366af4e60df295b7fcaed3b06"
Oct 30 20:54:32.909872 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:32.909852    5071 scope.go:122] "RemoveContainer" containerID="354e883b37c18432ea093778523ead22341b2289c37a8e9050d5d21a6800d856"
Oct 30 20:54:32.919517 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:32.919475    5071 scope.go:122] "RemoveContainer" containerID="f37ccb8ca15b318706c0c7fd8e5bd6142730a5c4712156d5fa0a73f0ba1ff44f"
Oct 30 20:54:32.928807 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:32.928771    5071 scope.go:122] "RemoveContainer" containerID="5084c31899b706da6dea4857528ce6728bbd5b9030660b4652b22645fcb2419f"
Oct 30 20:54:32.939025 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:32.939006    5071 scope.go:122] "RemoveContainer" containerID="1ffb0dee5cf453e0d5e975979e60c7fffe1d165a05f0126f22a9b3c8030548ec"
Oct 30 20:54:32.949223 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:32.949200    5071 scope.go:122] "RemoveContainer" containerID="93e4c63fd0cc4b8d0a55600b38159631c4bc474ba48db35f878ba6c2f8a357cc"
Oct 30 20:54:32.960266 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:32.960247    5071 scope.go:122] "RemoveContainer" containerID="b7f3f344358e8524aeeac5a775d512042c711dcd3691fb4fc4a5174ca24cc58f"
Oct 30 20:54:32.999928 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:32.999903    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d9b3f6-86cfc8c66c-l48wg"]
Oct 30 20:54:33.010373 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:54:33.010346    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940a58d0_5deb_4bc9_8832_5a17fa300539.slice/crio-582a789d3aea6cc26912c29e8bc1353c2853cdd7e1084cc650fbcff87a13c5d0 WatchSource:0}: Error finding container 582a789d3aea6cc26912c29e8bc1353c2853cdd7e1084cc650fbcff87a13c5d0: Status 404 returned error can't find the container with id 582a789d3aea6cc26912c29e8bc1353c2853cdd7e1084cc650fbcff87a13c5d0
Oct 30 20:54:33.025917 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:33.025891    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"]
Oct 30 20:54:33.026243 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:33.026217    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t" podUID="00b1175c-ea5d-49a8-b721-93fd709970c1" containerName="istio-proxy" containerID="cri-o://15ede708ebc06744572625fcfd27ff799d5c51483946b49ccf1e095a9db1eab4" gracePeriod=30
Oct 30 20:54:33.050391 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:33.050360    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"]
Oct 30 20:54:33.050630 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:33.050603    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm" podUID="903e1c2f-42e8-4702-8368-3831f717fd8e" containerName="istio-proxy" containerID="cri-o://5ceeab6e98675cc7f87f3a1923620bc5c46f692c3b6077a80eeccfc61a22b189" gracePeriod=30
Oct 30 20:54:33.078705 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:33.078679    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"]
Oct 30 20:54:33.079189 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:33.078950    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744" podUID="84cb8a9e-b4d1-4c1b-9338-2ea77abec886" containerName="istio-proxy" containerID="cri-o://01f8afa45021133acda78d054117a722b14b334a8ed629371c812a4748bf5a3d" gracePeriod=30
Oct 30 20:54:33.087387 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:33.087004    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"]
Oct 30 20:54:33.087387 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:33.087225    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5" podUID="b8d10219-d907-47b1-be66-70a4e9d40465" containerName="istio-proxy" containerID="cri-o://a900003585f1a457a8d53a69adc4c1f492f4d44f31ebb02cefb7fff6ac2c2183" gracePeriod=30
Oct 30 20:54:33.096392 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:33.096208    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-d9b3f6-845694f564-8jdjx"]
Oct 30 20:54:33.102178 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:33.102152    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"]
Oct 30 20:54:33.102440 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:33.102409    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92" podUID="31b52815-7184-4c7b-a294-a2d9a7d0c099" containerName="istio-proxy" containerID="cri-o://67664ec9a132ab24151dca469320d3494a4f9480c3c446b816cd116e47234e06" gracePeriod=30
Oct 30 20:54:33.108761 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:33.108736    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"]
Oct 30 20:54:33.109005 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:33.108976    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g" podUID="05a72fc0-2554-4f73-935e-ade8fb962a7c" containerName="istio-proxy" containerID="cri-o://0d30aebc1f53767b2a8c9a23e00404dd3b977c5126fc9e26b5bc757dd3361f91" gracePeriod=30
Oct 30 20:54:33.129969 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:33.128870    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"]
Oct 30 20:54:33.129969 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:33.129785    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj" podUID="09665d83-c120-4c47-bc5f-45fa16149bb4" containerName="istio-proxy" containerID="cri-o://bf58317edd0ff572f75077ded61beb7eec136964bb1ece6e5c84a6a87cd40869" gracePeriod=30
Oct 30 20:54:33.147323 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:33.147292    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"]
Oct 30 20:54:33.147795 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:33.147762    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47" podUID="736322c2-4cdb-4fd1-bbc1-15c089d827d9" containerName="istio-proxy" containerID="cri-o://2d13fba3913f9425b772b991272a49bbf4d1d0054558e2b5b5de3a4159d5fe4c" gracePeriod=30
Oct 30 20:54:33.182565 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:33.182123    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"]
Oct 30 20:54:33.182565 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:33.182353    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb" podUID="5cdad370-2ff5-435c-bbfe-2ba61a325d47" containerName="istio-proxy" containerID="cri-o://22a7dc2536154ac2157046ebc9b68adc172c85397eb9352cfc893e481db56048" gracePeriod=30
Oct 30 20:54:33.189928 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:33.184174    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"]
Oct 30 20:54:33.189928 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:33.184364    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87" podUID="723f1905-6d5a-45a7-a2bb-d5753a0fda9c" containerName="istio-proxy" containerID="cri-o://b3689f02a0c436ed2add7966e749d0c6f7cc617b2bad25d8452f9e7bf29e240b" gracePeriod=30
Oct 30 20:54:33.191249 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:33.190903    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"]
Oct 30 20:54:33.191249 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:33.191109    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s" podUID="fd81aa48-0f88-44fb-8baa-db1deec4a2df" containerName="istio-proxy" containerID="cri-o://753e257909c6531af48f1cafec5c39fb421f9e97fede074c88bf601b9071eb0f" gracePeriod=30
Oct 30 20:54:33.197246 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:33.196762    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"]
Oct 30 20:54:33.197246 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:33.197036    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl" podUID="aa5037cf-aea2-403e-9865-8df55fe7e04f" containerName="istio-proxy" containerID="cri-o://4dab2753431dab7f9c6ebd9aae38bb413c72553660c8459a4ab0db453edc091a" gracePeriod=30
Oct 30 20:54:33.256903 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:33.256869    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d9b3f6-86cfc8c66c-l48wg" event={"ID":"940a58d0-5deb-4bc9-8832-5a17fa300539","Type":"ContainerStarted","Data":"a3dab290335c6e2959d6be89508aa8433cd6ee89977f3d0b52986fb0abd5c7d8"}
Oct 30 20:54:33.257114 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:33.257061    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d9b3f6-86cfc8c66c-l48wg" event={"ID":"940a58d0-5deb-4bc9-8832-5a17fa300539","Type":"ContainerStarted","Data":"582a789d3aea6cc26912c29e8bc1353c2853cdd7e1084cc650fbcff87a13c5d0"}
Oct 30 20:54:33.257445 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:33.257422    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d9b3f6-86cfc8c66c-l48wg"
Oct 30 20:54:33.258905 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:33.258878    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d9b3f6-845694f564-8jdjx" event={"ID":"902523f4-4967-4ef2-97be-632d3a0c6383","Type":"ContainerStarted","Data":"f3498ceb33be705edde86b807f7bb73afd8df3bd7b090206fb40fa0197731f64"}
Oct 30 20:54:33.617847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:33.617448    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:33.866807 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:33.866771    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:54:34.280226 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:34.280187    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d9b3f6-845694f564-8jdjx" event={"ID":"902523f4-4967-4ef2-97be-632d3a0c6383","Type":"ContainerStarted","Data":"8f0bf82524098ef28f1ccc1e285f05d2c56e00c2eaae032e2e5af1163ac6f331"}
Oct 30 20:54:34.280416 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:34.280282    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d9b3f6-845694f564-8jdjx"
Oct 30 20:54:34.352532 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:34.352479    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d9b3f6-86cfc8c66c-l48wg" podStartSLOduration=3.3524656840000002 podStartE2EDuration="3.352465684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:34.351921835 +0000 UTC m=+1686.268943373" watchObservedRunningTime="2026-10-30 20:54:34.352465684 +0000 UTC m=+1686.269487221"
Oct 30 20:54:34.961230 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:34.961173    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d9b3f6-845694f564-8jdjx" podStartSLOduration=3.961158532 podStartE2EDuration="3.961158532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:34.95529689 +0000 UTC m=+1686.872318426" watchObservedRunningTime="2026-10-30 20:54:34.961158532 +0000 UTC m=+1686.878180068"
Oct 30 20:54:35.388100 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:35.388065    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:54:35.401055 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:35.400688    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47" podUID="736322c2-4cdb-4fd1-bbc1-15c089d827d9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:54:35.449380 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:35.449340    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:35.622672 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:35.622612    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:35.744568 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:35.744482    5071 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:54:35.744568 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:35.744543    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t" podUID="00b1175c-ea5d-49a8-b721-93fd709970c1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:54:36.280526 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:36.280492    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:36.281695 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:36.281645    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:54:36.281851 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:36.281725    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g" podUID="05a72fc0-2554-4f73-935e-ade8fb962a7c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:54:36.544096 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:36.544059    5071 patch_prober.go:29] interesting pod/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:54:36.544241 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:36.544110    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87" podUID="723f1905-6d5a-45a7-a2bb-d5753a0fda9c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:54:36.919400 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:36.917988    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-15199a-69b969cbfb-whtbl"
Oct 30 20:54:36.923137 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:36.923112    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-15199a-79b8f8b875-m8l5v"
Oct 30 20:54:37.966642 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:37.966601    5071 patch_prober.go:29] interesting pod/e2e-worker-w4-395022-openshift-default-559878f94c-v9744 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:54:37.967073 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:37.966659    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744" podUID="84cb8a9e-b4d1-4c1b-9338-2ea77abec886" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:54:38.006188 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.006149    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:38.208151 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.203680    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:38.400100 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.400065    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:38.477769 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.477498    5071 generic.go:356] "Generic (PLEG): container finished" podID="09665d83-c120-4c47-bc5f-45fa16149bb4" containerID="bf58317edd0ff572f75077ded61beb7eec136964bb1ece6e5c84a6a87cd40869" exitCode=0
Oct 30 20:54:38.477769 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.477619    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj" event={"ID":"09665d83-c120-4c47-bc5f-45fa16149bb4","Type":"ContainerDied","Data":"bf58317edd0ff572f75077ded61beb7eec136964bb1ece6e5c84a6a87cd40869"}
Oct 30 20:54:38.503536 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.503207    5071 generic.go:356] "Generic (PLEG): container finished" podID="31b52815-7184-4c7b-a294-a2d9a7d0c099" containerID="67664ec9a132ab24151dca469320d3494a4f9480c3c446b816cd116e47234e06" exitCode=0
Oct 30 20:54:38.503536 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.503317    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92" event={"ID":"31b52815-7184-4c7b-a294-a2d9a7d0c099","Type":"ContainerDied","Data":"67664ec9a132ab24151dca469320d3494a4f9480c3c446b816cd116e47234e06"}
Oct 30 20:54:38.506201 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.506171    5071 generic.go:356] "Generic (PLEG): container finished" podID="b8d10219-d907-47b1-be66-70a4e9d40465" containerID="a900003585f1a457a8d53a69adc4c1f492f4d44f31ebb02cefb7fff6ac2c2183" exitCode=0
Oct 30 20:54:38.506508 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.506468    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5" event={"ID":"b8d10219-d907-47b1-be66-70a4e9d40465","Type":"ContainerDied","Data":"a900003585f1a457a8d53a69adc4c1f492f4d44f31ebb02cefb7fff6ac2c2183"}
Oct 30 20:54:38.525834 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.525777    5071 patch_prober.go:29] interesting pod/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 20:54:38.526117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.526068    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj" podUID="09665d83-c120-4c47-bc5f-45fa16149bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 20:54:38.581179 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.579186    5071 generic.go:356] "Generic (PLEG): container finished" podID="00b1175c-ea5d-49a8-b721-93fd709970c1" containerID="15ede708ebc06744572625fcfd27ff799d5c51483946b49ccf1e095a9db1eab4" exitCode=0
Oct 30 20:54:38.581179 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.579642    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t" event={"ID":"00b1175c-ea5d-49a8-b721-93fd709970c1","Type":"ContainerDied","Data":"15ede708ebc06744572625fcfd27ff799d5c51483946b49ccf1e095a9db1eab4"}
Oct 30 20:54:38.605494 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.601173    5071 generic.go:356] "Generic (PLEG): container finished" podID="903e1c2f-42e8-4702-8368-3831f717fd8e" containerID="5ceeab6e98675cc7f87f3a1923620bc5c46f692c3b6077a80eeccfc61a22b189" exitCode=0
Oct 30 20:54:38.605494 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.601337    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm" event={"ID":"903e1c2f-42e8-4702-8368-3831f717fd8e","Type":"ContainerDied","Data":"5ceeab6e98675cc7f87f3a1923620bc5c46f692c3b6077a80eeccfc61a22b189"}
Oct 30 20:54:38.613363 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.612567    5071 generic.go:356] "Generic (PLEG): container finished" podID="736322c2-4cdb-4fd1-bbc1-15c089d827d9" containerID="2d13fba3913f9425b772b991272a49bbf4d1d0054558e2b5b5de3a4159d5fe4c" exitCode=0
Oct 30 20:54:38.613363 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.612758    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47" event={"ID":"736322c2-4cdb-4fd1-bbc1-15c089d827d9","Type":"ContainerDied","Data":"2d13fba3913f9425b772b991272a49bbf4d1d0054558e2b5b5de3a4159d5fe4c"}
Oct 30 20:54:38.617644 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.617615    5071 generic.go:356] "Generic (PLEG): container finished" podID="84cb8a9e-b4d1-4c1b-9338-2ea77abec886" containerID="01f8afa45021133acda78d054117a722b14b334a8ed629371c812a4748bf5a3d" exitCode=0
Oct 30 20:54:38.617783 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.617764    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744" event={"ID":"84cb8a9e-b4d1-4c1b-9338-2ea77abec886","Type":"ContainerDied","Data":"01f8afa45021133acda78d054117a722b14b334a8ed629371c812a4748bf5a3d"}
Oct 30 20:54:38.635368 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.630504    5071 generic.go:356] "Generic (PLEG): container finished" podID="5cdad370-2ff5-435c-bbfe-2ba61a325d47" containerID="22a7dc2536154ac2157046ebc9b68adc172c85397eb9352cfc893e481db56048" exitCode=0
Oct 30 20:54:38.635368 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.630608    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb" event={"ID":"5cdad370-2ff5-435c-bbfe-2ba61a325d47","Type":"ContainerDied","Data":"22a7dc2536154ac2157046ebc9b68adc172c85397eb9352cfc893e481db56048"}
Oct 30 20:54:38.660005 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.659567    5071 generic.go:356] "Generic (PLEG): container finished" podID="05a72fc0-2554-4f73-935e-ade8fb962a7c" containerID="0d30aebc1f53767b2a8c9a23e00404dd3b977c5126fc9e26b5bc757dd3361f91" exitCode=0
Oct 30 20:54:38.660005 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.659750    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g" event={"ID":"05a72fc0-2554-4f73-935e-ade8fb962a7c","Type":"ContainerDied","Data":"0d30aebc1f53767b2a8c9a23e00404dd3b977c5126fc9e26b5bc757dd3361f91"}
Oct 30 20:54:38.663312 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.662801    5071 generic.go:356] "Generic (PLEG): container finished" podID="fd81aa48-0f88-44fb-8baa-db1deec4a2df" containerID="753e257909c6531af48f1cafec5c39fb421f9e97fede074c88bf601b9071eb0f" exitCode=0
Oct 30 20:54:38.663312 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.662901    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s" event={"ID":"fd81aa48-0f88-44fb-8baa-db1deec4a2df","Type":"ContainerDied","Data":"753e257909c6531af48f1cafec5c39fb421f9e97fede074c88bf601b9071eb0f"}
Oct 30 20:54:38.666218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.665780    5071 generic.go:356] "Generic (PLEG): container finished" podID="723f1905-6d5a-45a7-a2bb-d5753a0fda9c" containerID="b3689f02a0c436ed2add7966e749d0c6f7cc617b2bad25d8452f9e7bf29e240b" exitCode=0
Oct 30 20:54:38.666218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.665847    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87" event={"ID":"723f1905-6d5a-45a7-a2bb-d5753a0fda9c","Type":"ContainerDied","Data":"b3689f02a0c436ed2add7966e749d0c6f7cc617b2bad25d8452f9e7bf29e240b"}
Oct 30 20:54:38.670176 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.669876    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"
Oct 30 20:54:38.702920 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.701832    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"
Oct 30 20:54:38.743315 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.743203    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31b52815-7184-4c7b-a294-a2d9a7d0c099-kube-api-access-4wzgt\" (UniqueName: \"kubernetes.io/projected/31b52815-7184-4c7b-a294-a2d9a7d0c099-kube-api-access-4wzgt\") pod \"31b52815-7184-4c7b-a294-a2d9a7d0c099\" (UID: \"31b52815-7184-4c7b-a294-a2d9a7d0c099\") "
Oct 30 20:54:38.743315 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.743304    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b52815-7184-4c7b-a294-a2d9a7d0c099-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b52815-7184-4c7b-a294-a2d9a7d0c099-credential-socket\") pod \"31b52815-7184-4c7b-a294-a2d9a7d0c099\" (UID: \"31b52815-7184-4c7b-a294-a2d9a7d0c099\") "
Oct 30 20:54:38.743623 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.743375    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b52815-7184-4c7b-a294-a2d9a7d0c099-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b52815-7184-4c7b-a294-a2d9a7d0c099-istio-envoy\") pod \"31b52815-7184-4c7b-a294-a2d9a7d0c099\" (UID: \"31b52815-7184-4c7b-a294-a2d9a7d0c099\") "
Oct 30 20:54:38.743623 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.743450    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31b52815-7184-4c7b-a294-a2d9a7d0c099-istio-token\" (UniqueName: \"kubernetes.io/projected/31b52815-7184-4c7b-a294-a2d9a7d0c099-istio-token\") pod \"31b52815-7184-4c7b-a294-a2d9a7d0c099\" (UID: \"31b52815-7184-4c7b-a294-a2d9a7d0c099\") "
Oct 30 20:54:38.743623 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.743492    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b52815-7184-4c7b-a294-a2d9a7d0c099-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b52815-7184-4c7b-a294-a2d9a7d0c099-workload-certs\") pod \"31b52815-7184-4c7b-a294-a2d9a7d0c099\" (UID: \"31b52815-7184-4c7b-a294-a2d9a7d0c099\") "
Oct 30 20:54:38.743623 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.743572    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b52815-7184-4c7b-a294-a2d9a7d0c099-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b52815-7184-4c7b-a294-a2d9a7d0c099-workload-socket\") pod \"31b52815-7184-4c7b-a294-a2d9a7d0c099\" (UID: \"31b52815-7184-4c7b-a294-a2d9a7d0c099\") "
Oct 30 20:54:38.743869 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.743679    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31b52815-7184-4c7b-a294-a2d9a7d0c099-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b52815-7184-4c7b-a294-a2d9a7d0c099-istio-podinfo\") pod \"31b52815-7184-4c7b-a294-a2d9a7d0c099\" (UID: \"31b52815-7184-4c7b-a294-a2d9a7d0c099\") "
Oct 30 20:54:38.743869 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.743804    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31b52815-7184-4c7b-a294-a2d9a7d0c099-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b52815-7184-4c7b-a294-a2d9a7d0c099-istiod-ca-cert\") pod \"31b52815-7184-4c7b-a294-a2d9a7d0c099\" (UID: \"31b52815-7184-4c7b-a294-a2d9a7d0c099\") "
Oct 30 20:54:38.743972 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.743920    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31b52815-7184-4c7b-a294-a2d9a7d0c099-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b52815-7184-4c7b-a294-a2d9a7d0c099-istio-data\") pod \"31b52815-7184-4c7b-a294-a2d9a7d0c099\" (UID: \"31b52815-7184-4c7b-a294-a2d9a7d0c099\") "
Oct 30 20:54:38.744644 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.744618    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b52815-7184-4c7b-a294-a2d9a7d0c099-istio-data" pod "31b52815-7184-4c7b-a294-a2d9a7d0c099" (UID: "31b52815-7184-4c7b-a294-a2d9a7d0c099"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:38.745151 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.745129    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b52815-7184-4c7b-a294-a2d9a7d0c099-workload-certs" pod "31b52815-7184-4c7b-a294-a2d9a7d0c099" (UID: "31b52815-7184-4c7b-a294-a2d9a7d0c099"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:38.745307 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.745290    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b52815-7184-4c7b-a294-a2d9a7d0c099-credential-socket" pod "31b52815-7184-4c7b-a294-a2d9a7d0c099" (UID: "31b52815-7184-4c7b-a294-a2d9a7d0c099"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:38.746276 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.746205    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b52815-7184-4c7b-a294-a2d9a7d0c099-workload-socket" pod "31b52815-7184-4c7b-a294-a2d9a7d0c099" (UID: "31b52815-7184-4c7b-a294-a2d9a7d0c099"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:38.746372 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.746346    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31b52815-7184-4c7b-a294-a2d9a7d0c099-istiod-ca-cert" pod "31b52815-7184-4c7b-a294-a2d9a7d0c099" (UID: "31b52815-7184-4c7b-a294-a2d9a7d0c099"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:38.748400 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.748360    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31b52815-7184-4c7b-a294-a2d9a7d0c099-istio-envoy" pod "31b52815-7184-4c7b-a294-a2d9a7d0c099" (UID: "31b52815-7184-4c7b-a294-a2d9a7d0c099"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:38.750253 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.750228    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31b52815-7184-4c7b-a294-a2d9a7d0c099-istio-podinfo" pod "31b52815-7184-4c7b-a294-a2d9a7d0c099" (UID: "31b52815-7184-4c7b-a294-a2d9a7d0c099"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:54:38.751181 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.751161    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31b52815-7184-4c7b-a294-a2d9a7d0c099-kube-api-access-4wzgt" pod "31b52815-7184-4c7b-a294-a2d9a7d0c099" (UID: "31b52815-7184-4c7b-a294-a2d9a7d0c099"). InnerVolumeSpecName "kube-api-access-4wzgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:38.751928 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.751907    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31b52815-7184-4c7b-a294-a2d9a7d0c099-istio-token" pod "31b52815-7184-4c7b-a294-a2d9a7d0c099" (UID: "31b52815-7184-4c7b-a294-a2d9a7d0c099"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:38.802077 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.802050    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"
Oct 30 20:54:38.829200 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.829172    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"
Oct 30 20:54:38.845663 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.845635    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00b1175c-ea5d-49a8-b721-93fd709970c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00b1175c-ea5d-49a8-b721-93fd709970c1-istio-data\") pod \"00b1175c-ea5d-49a8-b721-93fd709970c1\" (UID: \"00b1175c-ea5d-49a8-b721-93fd709970c1\") "
Oct 30 20:54:38.848666 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.848639    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00b1175c-ea5d-49a8-b721-93fd709970c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00b1175c-ea5d-49a8-b721-93fd709970c1-istio-envoy\") pod \"00b1175c-ea5d-49a8-b721-93fd709970c1\" (UID: \"00b1175c-ea5d-49a8-b721-93fd709970c1\") "
Oct 30 20:54:38.848938 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.848916    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00b1175c-ea5d-49a8-b721-93fd709970c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b1175c-ea5d-49a8-b721-93fd709970c1-workload-socket\") pod \"00b1175c-ea5d-49a8-b721-93fd709970c1\" (UID: \"00b1175c-ea5d-49a8-b721-93fd709970c1\") "
Oct 30 20:54:38.849104 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.849087    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00b1175c-ea5d-49a8-b721-93fd709970c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b1175c-ea5d-49a8-b721-93fd709970c1-credential-socket\") pod \"00b1175c-ea5d-49a8-b721-93fd709970c1\" (UID: \"00b1175c-ea5d-49a8-b721-93fd709970c1\") "
Oct 30 20:54:38.849257 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.849241    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00b1175c-ea5d-49a8-b721-93fd709970c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00b1175c-ea5d-49a8-b721-93fd709970c1-workload-certs\") pod \"00b1175c-ea5d-49a8-b721-93fd709970c1\" (UID: \"00b1175c-ea5d-49a8-b721-93fd709970c1\") "
Oct 30 20:54:38.849354 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.849274    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00b1175c-ea5d-49a8-b721-93fd709970c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00b1175c-ea5d-49a8-b721-93fd709970c1-istio-podinfo\") pod \"00b1175c-ea5d-49a8-b721-93fd709970c1\" (UID: \"00b1175c-ea5d-49a8-b721-93fd709970c1\") "
Oct 30 20:54:38.849410 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.849381    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00b1175c-ea5d-49a8-b721-93fd709970c1-kube-api-access-vh8fl\" (UniqueName: \"kubernetes.io/projected/00b1175c-ea5d-49a8-b721-93fd709970c1-kube-api-access-vh8fl\") pod \"00b1175c-ea5d-49a8-b721-93fd709970c1\" (UID: \"00b1175c-ea5d-49a8-b721-93fd709970c1\") "
Oct 30 20:54:38.849490 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.849474    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00b1175c-ea5d-49a8-b721-93fd709970c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00b1175c-ea5d-49a8-b721-93fd709970c1-istiod-ca-cert\") pod \"00b1175c-ea5d-49a8-b721-93fd709970c1\" (UID: \"00b1175c-ea5d-49a8-b721-93fd709970c1\") "
Oct 30 20:54:38.849571 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.849555    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00b1175c-ea5d-49a8-b721-93fd709970c1-istio-token\" (UniqueName: \"kubernetes.io/projected/00b1175c-ea5d-49a8-b721-93fd709970c1-istio-token\") pod \"00b1175c-ea5d-49a8-b721-93fd709970c1\" (UID: \"00b1175c-ea5d-49a8-b721-93fd709970c1\") "
Oct 30 20:54:38.850208 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.850187    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31b52815-7184-4c7b-a294-a2d9a7d0c099-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:38.850313 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.850212    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31b52815-7184-4c7b-a294-a2d9a7d0c099-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:38.850313 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.850227    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wzgt\" (UniqueName: \"kubernetes.io/projected/31b52815-7184-4c7b-a294-a2d9a7d0c099-kube-api-access-4wzgt\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:38.850313 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.850242    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b52815-7184-4c7b-a294-a2d9a7d0c099-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:38.850313 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.850255    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31b52815-7184-4c7b-a294-a2d9a7d0c099-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:38.850313 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.850268    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31b52815-7184-4c7b-a294-a2d9a7d0c099-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:38.850313 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.850282    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31b52815-7184-4c7b-a294-a2d9a7d0c099-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:38.850313 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.850295    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31b52815-7184-4c7b-a294-a2d9a7d0c099-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:38.850313 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.850308    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31b52815-7184-4c7b-a294-a2d9a7d0c099-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:38.851040 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.848148    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b1175c-ea5d-49a8-b721-93fd709970c1-istio-data" pod "00b1175c-ea5d-49a8-b721-93fd709970c1" (UID: "00b1175c-ea5d-49a8-b721-93fd709970c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:38.851188 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.850429    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b1175c-ea5d-49a8-b721-93fd709970c1-workload-socket" pod "00b1175c-ea5d-49a8-b721-93fd709970c1" (UID: "00b1175c-ea5d-49a8-b721-93fd709970c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:38.851253 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.851215    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b1175c-ea5d-49a8-b721-93fd709970c1-credential-socket" pod "00b1175c-ea5d-49a8-b721-93fd709970c1" (UID: "00b1175c-ea5d-49a8-b721-93fd709970c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:38.852220 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.852162    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00b1175c-ea5d-49a8-b721-93fd709970c1-istiod-ca-cert" pod "00b1175c-ea5d-49a8-b721-93fd709970c1" (UID: "00b1175c-ea5d-49a8-b721-93fd709970c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:38.852220 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.852197    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b1175c-ea5d-49a8-b721-93fd709970c1-workload-certs" pod "00b1175c-ea5d-49a8-b721-93fd709970c1" (UID: "00b1175c-ea5d-49a8-b721-93fd709970c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:38.855313 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.855270    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00b1175c-ea5d-49a8-b721-93fd709970c1-kube-api-access-vh8fl" pod "00b1175c-ea5d-49a8-b721-93fd709970c1" (UID: "00b1175c-ea5d-49a8-b721-93fd709970c1"). InnerVolumeSpecName "kube-api-access-vh8fl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:38.856108 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.856087    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00b1175c-ea5d-49a8-b721-93fd709970c1-istio-token" pod "00b1175c-ea5d-49a8-b721-93fd709970c1" (UID: "00b1175c-ea5d-49a8-b721-93fd709970c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:38.857757 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.857723    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00b1175c-ea5d-49a8-b721-93fd709970c1-istio-podinfo" pod "00b1175c-ea5d-49a8-b721-93fd709970c1" (UID: "00b1175c-ea5d-49a8-b721-93fd709970c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:54:38.865168 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.865139    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b1175c-ea5d-49a8-b721-93fd709970c1-istio-envoy" pod "00b1175c-ea5d-49a8-b721-93fd709970c1" (UID: "00b1175c-ea5d-49a8-b721-93fd709970c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:38.876493 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.876295    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"
Oct 30 20:54:38.882018 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.878625    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"
Oct 30 20:54:38.889257 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.889228    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"
Oct 30 20:54:38.896082 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.896060    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"
Oct 30 20:54:38.951465 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.951293    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09665d83-c120-4c47-bc5f-45fa16149bb4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09665d83-c120-4c47-bc5f-45fa16149bb4-istio-podinfo\") pod \"09665d83-c120-4c47-bc5f-45fa16149bb4\" (UID: \"09665d83-c120-4c47-bc5f-45fa16149bb4\") "
Oct 30 20:54:38.951465 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.951355    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/903e1c2f-42e8-4702-8368-3831f717fd8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/903e1c2f-42e8-4702-8368-3831f717fd8e-workload-socket\") pod \"903e1c2f-42e8-4702-8368-3831f717fd8e\" (UID: \"903e1c2f-42e8-4702-8368-3831f717fd8e\") "
Oct 30 20:54:38.951465 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.951380    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/903e1c2f-42e8-4702-8368-3831f717fd8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/903e1c2f-42e8-4702-8368-3831f717fd8e-istiod-ca-cert\") pod \"903e1c2f-42e8-4702-8368-3831f717fd8e\" (UID: \"903e1c2f-42e8-4702-8368-3831f717fd8e\") "
Oct 30 20:54:38.951465 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.951423    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-kube-api-access-hhq26\" (UniqueName: \"kubernetes.io/projected/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-kube-api-access-hhq26\") pod \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\" (UID: \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\") "
Oct 30 20:54:38.952154 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.951931    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8d10219-d907-47b1-be66-70a4e9d40465-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8d10219-d907-47b1-be66-70a4e9d40465-workload-certs\") pod \"b8d10219-d907-47b1-be66-70a4e9d40465\" (UID: \"b8d10219-d907-47b1-be66-70a4e9d40465\") "
Oct 30 20:54:38.952154 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.952017    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8d10219-d907-47b1-be66-70a4e9d40465-istio-token\" (UniqueName: \"kubernetes.io/projected/b8d10219-d907-47b1-be66-70a4e9d40465-istio-token\") pod \"b8d10219-d907-47b1-be66-70a4e9d40465\" (UID: \"b8d10219-d907-47b1-be66-70a4e9d40465\") "
Oct 30 20:54:38.952862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.952236    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09665d83-c120-4c47-bc5f-45fa16149bb4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09665d83-c120-4c47-bc5f-45fa16149bb4-istiod-ca-cert\") pod \"09665d83-c120-4c47-bc5f-45fa16149bb4\" (UID: \"09665d83-c120-4c47-bc5f-45fa16149bb4\") "
Oct 30 20:54:38.952862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.952268    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8d10219-d907-47b1-be66-70a4e9d40465-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d10219-d907-47b1-be66-70a4e9d40465-workload-socket\") pod \"b8d10219-d907-47b1-be66-70a4e9d40465\" (UID: \"b8d10219-d907-47b1-be66-70a4e9d40465\") "
Oct 30 20:54:38.952862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.952302    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/903e1c2f-42e8-4702-8368-3831f717fd8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/903e1c2f-42e8-4702-8368-3831f717fd8e-istio-data\") pod \"903e1c2f-42e8-4702-8368-3831f717fd8e\" (UID: \"903e1c2f-42e8-4702-8368-3831f717fd8e\") "
Oct 30 20:54:38.952862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.952331    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/903e1c2f-42e8-4702-8368-3831f717fd8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/903e1c2f-42e8-4702-8368-3831f717fd8e-istio-podinfo\") pod \"903e1c2f-42e8-4702-8368-3831f717fd8e\" (UID: \"903e1c2f-42e8-4702-8368-3831f717fd8e\") "
Oct 30 20:54:38.952862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.952353    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/903e1c2f-42e8-4702-8368-3831f717fd8e-kube-api-access-t6jjt\" (UniqueName: \"kubernetes.io/projected/903e1c2f-42e8-4702-8368-3831f717fd8e-kube-api-access-t6jjt\") pod \"903e1c2f-42e8-4702-8368-3831f717fd8e\" (UID: \"903e1c2f-42e8-4702-8368-3831f717fd8e\") "
Oct 30 20:54:38.952862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.952377    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-workload-socket\") pod \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\" (UID: \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\") "
Oct 30 20:54:38.952862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.952419    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-credential-socket\") pod \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\" (UID: \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\") "
Oct 30 20:54:38.952862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.952442    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8d10219-d907-47b1-be66-70a4e9d40465-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d10219-d907-47b1-be66-70a4e9d40465-credential-socket\") pod \"b8d10219-d907-47b1-be66-70a4e9d40465\" (UID: \"b8d10219-d907-47b1-be66-70a4e9d40465\") "
Oct 30 20:54:38.952862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.952463    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09665d83-c120-4c47-bc5f-45fa16149bb4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09665d83-c120-4c47-bc5f-45fa16149bb4-workload-certs\") pod \"09665d83-c120-4c47-bc5f-45fa16149bb4\" (UID: \"09665d83-c120-4c47-bc5f-45fa16149bb4\") "
Oct 30 20:54:38.952862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.952479    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09665d83-c120-4c47-bc5f-45fa16149bb4-istio-token\" (UniqueName: \"kubernetes.io/projected/09665d83-c120-4c47-bc5f-45fa16149bb4-istio-token\") pod \"09665d83-c120-4c47-bc5f-45fa16149bb4\" (UID: \"09665d83-c120-4c47-bc5f-45fa16149bb4\") "
Oct 30 20:54:38.952862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.952496    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09665d83-c120-4c47-bc5f-45fa16149bb4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09665d83-c120-4c47-bc5f-45fa16149bb4-istio-envoy\") pod \"09665d83-c120-4c47-bc5f-45fa16149bb4\" (UID: \"09665d83-c120-4c47-bc5f-45fa16149bb4\") "
Oct 30 20:54:38.952862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.952533    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-istio-data\") pod \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\" (UID: \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\") "
Oct 30 20:54:38.952862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.952556    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-istio-envoy\") pod \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\" (UID: \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\") "
Oct 30 20:54:38.952862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.952577    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/903e1c2f-42e8-4702-8368-3831f717fd8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/903e1c2f-42e8-4702-8368-3831f717fd8e-workload-certs\") pod \"903e1c2f-42e8-4702-8368-3831f717fd8e\" (UID: \"903e1c2f-42e8-4702-8368-3831f717fd8e\") "
Oct 30 20:54:38.952862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.952596    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09665d83-c120-4c47-bc5f-45fa16149bb4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09665d83-c120-4c47-bc5f-45fa16149bb4-istio-data\") pod \"09665d83-c120-4c47-bc5f-45fa16149bb4\" (UID: \"09665d83-c120-4c47-bc5f-45fa16149bb4\") "
Oct 30 20:54:38.952862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.952614    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09665d83-c120-4c47-bc5f-45fa16149bb4-kube-api-access-qg9ld\" (UniqueName: \"kubernetes.io/projected/09665d83-c120-4c47-bc5f-45fa16149bb4-kube-api-access-qg9ld\") pod \"09665d83-c120-4c47-bc5f-45fa16149bb4\" (UID: \"09665d83-c120-4c47-bc5f-45fa16149bb4\") "
Oct 30 20:54:38.952862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.952632    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8d10219-d907-47b1-be66-70a4e9d40465-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8d10219-d907-47b1-be66-70a4e9d40465-istio-envoy\") pod \"b8d10219-d907-47b1-be66-70a4e9d40465\" (UID: \"b8d10219-d907-47b1-be66-70a4e9d40465\") "
Oct 30 20:54:38.952862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.952655    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-workload-certs\") pod \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\" (UID: \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\") "
Oct 30 20:54:38.952862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.952709    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-istio-podinfo\") pod \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\" (UID: \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\") "
Oct 30 20:54:38.952862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.952733    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8d10219-d907-47b1-be66-70a4e9d40465-kube-api-access-7rfjt\" (UniqueName: \"kubernetes.io/projected/b8d10219-d907-47b1-be66-70a4e9d40465-kube-api-access-7rfjt\") pod \"b8d10219-d907-47b1-be66-70a4e9d40465\" (UID: \"b8d10219-d907-47b1-be66-70a4e9d40465\") "
Oct 30 20:54:38.952862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.952756    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-istio-token\" (UniqueName: \"kubernetes.io/projected/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-istio-token\") pod \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\" (UID: \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\") "
Oct 30 20:54:38.952862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.952799    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8d10219-d907-47b1-be66-70a4e9d40465-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8d10219-d907-47b1-be66-70a4e9d40465-istiod-ca-cert\") pod \"b8d10219-d907-47b1-be66-70a4e9d40465\" (UID: \"b8d10219-d907-47b1-be66-70a4e9d40465\") "
Oct 30 20:54:38.968577 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.954283    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/903e1c2f-42e8-4702-8368-3831f717fd8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/903e1c2f-42e8-4702-8368-3831f717fd8e-credential-socket\") pod \"903e1c2f-42e8-4702-8368-3831f717fd8e\" (UID: \"903e1c2f-42e8-4702-8368-3831f717fd8e\") "
Oct 30 20:54:38.968577 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.954344    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/903e1c2f-42e8-4702-8368-3831f717fd8e-istio-token\" (UniqueName: \"kubernetes.io/projected/903e1c2f-42e8-4702-8368-3831f717fd8e-istio-token\") pod \"903e1c2f-42e8-4702-8368-3831f717fd8e\" (UID: \"903e1c2f-42e8-4702-8368-3831f717fd8e\") "
Oct 30 20:54:38.968577 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.954376    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09665d83-c120-4c47-bc5f-45fa16149bb4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09665d83-c120-4c47-bc5f-45fa16149bb4-workload-socket\") pod \"09665d83-c120-4c47-bc5f-45fa16149bb4\" (UID: \"09665d83-c120-4c47-bc5f-45fa16149bb4\") "
Oct 30 20:54:38.968577 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.954398    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09665d83-c120-4c47-bc5f-45fa16149bb4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09665d83-c120-4c47-bc5f-45fa16149bb4-credential-socket\") pod \"09665d83-c120-4c47-bc5f-45fa16149bb4\" (UID: \"09665d83-c120-4c47-bc5f-45fa16149bb4\") "
Oct 30 20:54:38.968577 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.954446    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8d10219-d907-47b1-be66-70a4e9d40465-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8d10219-d907-47b1-be66-70a4e9d40465-istio-podinfo\") pod \"b8d10219-d907-47b1-be66-70a4e9d40465\" (UID: \"b8d10219-d907-47b1-be66-70a4e9d40465\") "
Oct 30 20:54:38.968577 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.954463    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/903e1c2f-42e8-4702-8368-3831f717fd8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/903e1c2f-42e8-4702-8368-3831f717fd8e-istio-envoy\") pod \"903e1c2f-42e8-4702-8368-3831f717fd8e\" (UID: \"903e1c2f-42e8-4702-8368-3831f717fd8e\") "
Oct 30 20:54:38.968577 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.954531    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-istiod-ca-cert\") pod \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\" (UID: \"84cb8a9e-b4d1-4c1b-9338-2ea77abec886\") "
Oct 30 20:54:38.968577 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.954555    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8d10219-d907-47b1-be66-70a4e9d40465-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8d10219-d907-47b1-be66-70a4e9d40465-istio-data\") pod \"b8d10219-d907-47b1-be66-70a4e9d40465\" (UID: \"b8d10219-d907-47b1-be66-70a4e9d40465\") "
Oct 30 20:54:38.968577 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.968462    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09665d83-c120-4c47-bc5f-45fa16149bb4-istio-envoy" pod "09665d83-c120-4c47-bc5f-45fa16149bb4" (UID: "09665d83-c120-4c47-bc5f-45fa16149bb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:38.970643 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.970413    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/903e1c2f-42e8-4702-8368-3831f717fd8e-istio-data" pod "903e1c2f-42e8-4702-8368-3831f717fd8e" (UID: "903e1c2f-42e8-4702-8368-3831f717fd8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:38.971038 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.970857    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-istio-data" pod "84cb8a9e-b4d1-4c1b-9338-2ea77abec886" (UID: "84cb8a9e-b4d1-4c1b-9338-2ea77abec886"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:38.971038 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.970865    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00b1175c-ea5d-49a8-b721-93fd709970c1-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:38.971038 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.970907    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09665d83-c120-4c47-bc5f-45fa16149bb4-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:38.971038 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.970923    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh8fl\" (UniqueName: \"kubernetes.io/projected/00b1175c-ea5d-49a8-b721-93fd709970c1-kube-api-access-vh8fl\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:38.971038 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.970940    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00b1175c-ea5d-49a8-b721-93fd709970c1-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:38.971038 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.970954    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00b1175c-ea5d-49a8-b721-93fd709970c1-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:38.971038 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.970968    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00b1175c-ea5d-49a8-b721-93fd709970c1-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:38.971038 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.970983    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00b1175c-ea5d-49a8-b721-93fd709970c1-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:38.971038 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.970997    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/903e1c2f-42e8-4702-8368-3831f717fd8e-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:38.972906 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.971069    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/903e1c2f-42e8-4702-8368-3831f717fd8e-workload-socket" pod "903e1c2f-42e8-4702-8368-3831f717fd8e" (UID: "903e1c2f-42e8-4702-8368-3831f717fd8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:38.972906 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.971379    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/903e1c2f-42e8-4702-8368-3831f717fd8e-istiod-ca-cert" pod "903e1c2f-42e8-4702-8368-3831f717fd8e" (UID: "903e1c2f-42e8-4702-8368-3831f717fd8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:38.973008 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.972989    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b1175c-ea5d-49a8-b721-93fd709970c1-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:38.973085 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.973018    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b1175c-ea5d-49a8-b721-93fd709970c1-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:38.973085 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.973036    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00b1175c-ea5d-49a8-b721-93fd709970c1-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:38.974538 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.974303    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d10219-d907-47b1-be66-70a4e9d40465-istiod-ca-cert" pod "b8d10219-d907-47b1-be66-70a4e9d40465" (UID: "b8d10219-d907-47b1-be66-70a4e9d40465"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:38.974986 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.974687    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-workload-certs" pod "84cb8a9e-b4d1-4c1b-9338-2ea77abec886" (UID: "84cb8a9e-b4d1-4c1b-9338-2ea77abec886"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:38.976697 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.976671    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d10219-d907-47b1-be66-70a4e9d40465-workload-certs" pod "b8d10219-d907-47b1-be66-70a4e9d40465" (UID: "b8d10219-d907-47b1-be66-70a4e9d40465"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:38.977050 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.977022    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-credential-socket" pod "84cb8a9e-b4d1-4c1b-9338-2ea77abec886" (UID: "84cb8a9e-b4d1-4c1b-9338-2ea77abec886"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:38.981464 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.981440    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-kube-api-access-hhq26" pod "84cb8a9e-b4d1-4c1b-9338-2ea77abec886" (UID: "84cb8a9e-b4d1-4c1b-9338-2ea77abec886"). InnerVolumeSpecName "kube-api-access-hhq26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:38.981639 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.981612    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d10219-d907-47b1-be66-70a4e9d40465-istio-envoy" pod "b8d10219-d907-47b1-be66-70a4e9d40465" (UID: "b8d10219-d907-47b1-be66-70a4e9d40465"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:38.981711 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.981663    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09665d83-c120-4c47-bc5f-45fa16149bb4-istiod-ca-cert" pod "09665d83-c120-4c47-bc5f-45fa16149bb4" (UID: "09665d83-c120-4c47-bc5f-45fa16149bb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:38.981906 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.981883    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/903e1c2f-42e8-4702-8368-3831f717fd8e-workload-certs" pod "903e1c2f-42e8-4702-8368-3831f717fd8e" (UID: "903e1c2f-42e8-4702-8368-3831f717fd8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:38.981976 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.981965    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d10219-d907-47b1-be66-70a4e9d40465-workload-socket" pod "b8d10219-d907-47b1-be66-70a4e9d40465" (UID: "b8d10219-d907-47b1-be66-70a4e9d40465"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:38.982107 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.982087    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09665d83-c120-4c47-bc5f-45fa16149bb4-istio-data" pod "09665d83-c120-4c47-bc5f-45fa16149bb4" (UID: "09665d83-c120-4c47-bc5f-45fa16149bb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:38.983004 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.982750    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d10219-d907-47b1-be66-70a4e9d40465-istio-token" pod "b8d10219-d907-47b1-be66-70a4e9d40465" (UID: "b8d10219-d907-47b1-be66-70a4e9d40465"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:38.983229 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.983176    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/903e1c2f-42e8-4702-8368-3831f717fd8e-credential-socket" pod "903e1c2f-42e8-4702-8368-3831f717fd8e" (UID: "903e1c2f-42e8-4702-8368-3831f717fd8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:38.984977 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.984955    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-istio-podinfo" pod "84cb8a9e-b4d1-4c1b-9338-2ea77abec886" (UID: "84cb8a9e-b4d1-4c1b-9338-2ea77abec886"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:54:38.986035 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.986012    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"
Oct 30 20:54:38.986556 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.986343    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/903e1c2f-42e8-4702-8368-3831f717fd8e-istio-token" pod "903e1c2f-42e8-4702-8368-3831f717fd8e" (UID: "903e1c2f-42e8-4702-8368-3831f717fd8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:38.987418 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.986697    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09665d83-c120-4c47-bc5f-45fa16149bb4-workload-socket" pod "09665d83-c120-4c47-bc5f-45fa16149bb4" (UID: "09665d83-c120-4c47-bc5f-45fa16149bb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:38.987418 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.986948    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09665d83-c120-4c47-bc5f-45fa16149bb4-kube-api-access-qg9ld" pod "09665d83-c120-4c47-bc5f-45fa16149bb4" (UID: "09665d83-c120-4c47-bc5f-45fa16149bb4"). InnerVolumeSpecName "kube-api-access-qg9ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:38.987418 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.987195    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-istiod-ca-cert" pod "84cb8a9e-b4d1-4c1b-9338-2ea77abec886" (UID: "84cb8a9e-b4d1-4c1b-9338-2ea77abec886"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:38.988107 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.987693    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09665d83-c120-4c47-bc5f-45fa16149bb4-credential-socket" pod "09665d83-c120-4c47-bc5f-45fa16149bb4" (UID: "09665d83-c120-4c47-bc5f-45fa16149bb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:38.988538 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.988229    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d10219-d907-47b1-be66-70a4e9d40465-istio-data" pod "b8d10219-d907-47b1-be66-70a4e9d40465" (UID: "b8d10219-d907-47b1-be66-70a4e9d40465"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:38.988538 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.988290    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-istio-envoy" pod "84cb8a9e-b4d1-4c1b-9338-2ea77abec886" (UID: "84cb8a9e-b4d1-4c1b-9338-2ea77abec886"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:38.988885 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.988847    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-workload-socket" pod "84cb8a9e-b4d1-4c1b-9338-2ea77abec886" (UID: "84cb8a9e-b4d1-4c1b-9338-2ea77abec886"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:38.989956 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.989192    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d10219-d907-47b1-be66-70a4e9d40465-kube-api-access-7rfjt" pod "b8d10219-d907-47b1-be66-70a4e9d40465" (UID: "b8d10219-d907-47b1-be66-70a4e9d40465"). InnerVolumeSpecName "kube-api-access-7rfjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:38.989956 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.989540    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09665d83-c120-4c47-bc5f-45fa16149bb4-workload-certs" pod "09665d83-c120-4c47-bc5f-45fa16149bb4" (UID: "09665d83-c120-4c47-bc5f-45fa16149bb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:38.990780 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.990671    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d10219-d907-47b1-be66-70a4e9d40465-credential-socket" pod "b8d10219-d907-47b1-be66-70a4e9d40465" (UID: "b8d10219-d907-47b1-be66-70a4e9d40465"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:38.990780 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.990738    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/903e1c2f-42e8-4702-8368-3831f717fd8e-istio-podinfo" pod "903e1c2f-42e8-4702-8368-3831f717fd8e" (UID: "903e1c2f-42e8-4702-8368-3831f717fd8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:54:38.994297 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.994195    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/903e1c2f-42e8-4702-8368-3831f717fd8e-kube-api-access-t6jjt" pod "903e1c2f-42e8-4702-8368-3831f717fd8e" (UID: "903e1c2f-42e8-4702-8368-3831f717fd8e"). InnerVolumeSpecName "kube-api-access-t6jjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:38.994297 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.994246    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8d10219-d907-47b1-be66-70a4e9d40465-istio-podinfo" pod "b8d10219-d907-47b1-be66-70a4e9d40465" (UID: "b8d10219-d907-47b1-be66-70a4e9d40465"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:54:38.996016 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.995981    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09665d83-c120-4c47-bc5f-45fa16149bb4-istio-podinfo" pod "09665d83-c120-4c47-bc5f-45fa16149bb4" (UID: "09665d83-c120-4c47-bc5f-45fa16149bb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:54:38.996515 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.996493    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/903e1c2f-42e8-4702-8368-3831f717fd8e-istio-envoy" pod "903e1c2f-42e8-4702-8368-3831f717fd8e" (UID: "903e1c2f-42e8-4702-8368-3831f717fd8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:38.996671 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.996653    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-istio-token" pod "84cb8a9e-b4d1-4c1b-9338-2ea77abec886" (UID: "84cb8a9e-b4d1-4c1b-9338-2ea77abec886"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:38.999729 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:38.999711    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"
Oct 30 20:54:39.002292 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.002270    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"
Oct 30 20:54:39.003268 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.003245    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09665d83-c120-4c47-bc5f-45fa16149bb4-istio-token" pod "09665d83-c120-4c47-bc5f-45fa16149bb4" (UID: "09665d83-c120-4c47-bc5f-45fa16149bb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.073978    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a72fc0-2554-4f73-935e-ade8fb962a7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a72fc0-2554-4f73-935e-ade8fb962a7c-workload-certs\") pod \"05a72fc0-2554-4f73-935e-ade8fb962a7c\" (UID: \"05a72fc0-2554-4f73-935e-ade8fb962a7c\") "
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.074166    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/736322c2-4cdb-4fd1-bbc1-15c089d827d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/736322c2-4cdb-4fd1-bbc1-15c089d827d9-workload-socket\") pod \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\" (UID: \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\") "
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.074207    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd81aa48-0f88-44fb-8baa-db1deec4a2df-istio-token\" (UniqueName: \"kubernetes.io/projected/fd81aa48-0f88-44fb-8baa-db1deec4a2df-istio-token\") pod \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\" (UID: \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\") "
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.074241    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/736322c2-4cdb-4fd1-bbc1-15c089d827d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/736322c2-4cdb-4fd1-bbc1-15c089d827d9-credential-socket\") pod \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\" (UID: \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\") "
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.074369    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/736322c2-4cdb-4fd1-bbc1-15c089d827d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/736322c2-4cdb-4fd1-bbc1-15c089d827d9-istio-data\") pod \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\" (UID: \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\") "
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.074402    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/736322c2-4cdb-4fd1-bbc1-15c089d827d9-kube-api-access-k4hh9\" (UniqueName: \"kubernetes.io/projected/736322c2-4cdb-4fd1-bbc1-15c089d827d9-kube-api-access-k4hh9\") pod \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\" (UID: \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\") "
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.074436    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/736322c2-4cdb-4fd1-bbc1-15c089d827d9-istio-token\" (UniqueName: \"kubernetes.io/projected/736322c2-4cdb-4fd1-bbc1-15c089d827d9-istio-token\") pod \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\" (UID: \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\") "
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.074490    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/736322c2-4cdb-4fd1-bbc1-15c089d827d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/736322c2-4cdb-4fd1-bbc1-15c089d827d9-istio-podinfo\") pod \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\" (UID: \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\") "
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.074528    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a72fc0-2554-4f73-935e-ade8fb962a7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05a72fc0-2554-4f73-935e-ade8fb962a7c-istio-envoy\") pod \"05a72fc0-2554-4f73-935e-ade8fb962a7c\" (UID: \"05a72fc0-2554-4f73-935e-ade8fb962a7c\") "
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.074586    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a72fc0-2554-4f73-935e-ade8fb962a7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a72fc0-2554-4f73-935e-ade8fb962a7c-credential-socket\") pod \"05a72fc0-2554-4f73-935e-ade8fb962a7c\" (UID: \"05a72fc0-2554-4f73-935e-ade8fb962a7c\") "
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.074627    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd81aa48-0f88-44fb-8baa-db1deec4a2df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd81aa48-0f88-44fb-8baa-db1deec4a2df-istio-podinfo\") pod \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\" (UID: \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\") "
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.074700    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05a72fc0-2554-4f73-935e-ade8fb962a7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05a72fc0-2554-4f73-935e-ade8fb962a7c-istiod-ca-cert\") pod \"05a72fc0-2554-4f73-935e-ade8fb962a7c\" (UID: \"05a72fc0-2554-4f73-935e-ade8fb962a7c\") "
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.074771    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a72fc0-2554-4f73-935e-ade8fb962a7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05a72fc0-2554-4f73-935e-ade8fb962a7c-istio-data\") pod \"05a72fc0-2554-4f73-935e-ade8fb962a7c\" (UID: \"05a72fc0-2554-4f73-935e-ade8fb962a7c\") "
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.074799    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05a72fc0-2554-4f73-935e-ade8fb962a7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05a72fc0-2554-4f73-935e-ade8fb962a7c-istio-podinfo\") pod \"05a72fc0-2554-4f73-935e-ade8fb962a7c\" (UID: \"05a72fc0-2554-4f73-935e-ade8fb962a7c\") "
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.074865    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd81aa48-0f88-44fb-8baa-db1deec4a2df-kube-api-access-hjxq5\" (UniqueName: \"kubernetes.io/projected/fd81aa48-0f88-44fb-8baa-db1deec4a2df-kube-api-access-hjxq5\") pod \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\" (UID: \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\") "
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.074913    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/736322c2-4cdb-4fd1-bbc1-15c089d827d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/736322c2-4cdb-4fd1-bbc1-15c089d827d9-workload-certs\") pod \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\" (UID: \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\") "
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.074977    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd81aa48-0f88-44fb-8baa-db1deec4a2df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd81aa48-0f88-44fb-8baa-db1deec4a2df-workload-socket\") pod \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\" (UID: \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\") "
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.075026    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/736322c2-4cdb-4fd1-bbc1-15c089d827d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/736322c2-4cdb-4fd1-bbc1-15c089d827d9-istiod-ca-cert\") pod \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\" (UID: \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\") "
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.075080    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05a72fc0-2554-4f73-935e-ade8fb962a7c-kube-api-access-q28mp\" (UniqueName: \"kubernetes.io/projected/05a72fc0-2554-4f73-935e-ade8fb962a7c-kube-api-access-q28mp\") pod \"05a72fc0-2554-4f73-935e-ade8fb962a7c\" (UID: \"05a72fc0-2554-4f73-935e-ade8fb962a7c\") "
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.075132    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd81aa48-0f88-44fb-8baa-db1deec4a2df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd81aa48-0f88-44fb-8baa-db1deec4a2df-workload-certs\") pod \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\" (UID: \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\") "
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.075211    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a72fc0-2554-4f73-935e-ade8fb962a7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a72fc0-2554-4f73-935e-ade8fb962a7c-workload-socket\") pod \"05a72fc0-2554-4f73-935e-ade8fb962a7c\" (UID: \"05a72fc0-2554-4f73-935e-ade8fb962a7c\") "
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.075243    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/736322c2-4cdb-4fd1-bbc1-15c089d827d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/736322c2-4cdb-4fd1-bbc1-15c089d827d9-istio-envoy\") pod \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\" (UID: \"736322c2-4cdb-4fd1-bbc1-15c089d827d9\") "
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.075344    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd81aa48-0f88-44fb-8baa-db1deec4a2df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd81aa48-0f88-44fb-8baa-db1deec4a2df-istiod-ca-cert\") pod \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\" (UID: \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\") "
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.075398    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd81aa48-0f88-44fb-8baa-db1deec4a2df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd81aa48-0f88-44fb-8baa-db1deec4a2df-istio-envoy\") pod \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\" (UID: \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\") "
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.075438    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd81aa48-0f88-44fb-8baa-db1deec4a2df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd81aa48-0f88-44fb-8baa-db1deec4a2df-credential-socket\") pod \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\" (UID: \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\") "
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.075497    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05a72fc0-2554-4f73-935e-ade8fb962a7c-istio-token\" (UniqueName: \"kubernetes.io/projected/05a72fc0-2554-4f73-935e-ade8fb962a7c-istio-token\") pod \"05a72fc0-2554-4f73-935e-ade8fb962a7c\" (UID: \"05a72fc0-2554-4f73-935e-ade8fb962a7c\") "
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.075538    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd81aa48-0f88-44fb-8baa-db1deec4a2df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd81aa48-0f88-44fb-8baa-db1deec4a2df-istio-data\") pod \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\" (UID: \"fd81aa48-0f88-44fb-8baa-db1deec4a2df\") "
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.076034    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a72fc0-2554-4f73-935e-ade8fb962a7c-istio-data" pod "05a72fc0-2554-4f73-935e-ade8fb962a7c" (UID: "05a72fc0-2554-4f73-935e-ade8fb962a7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.076232    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a72fc0-2554-4f73-935e-ade8fb962a7c-workload-certs" pod "05a72fc0-2554-4f73-935e-ade8fb962a7c" (UID: "05a72fc0-2554-4f73-935e-ade8fb962a7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.076405    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736322c2-4cdb-4fd1-bbc1-15c089d827d9-workload-socket" pod "736322c2-4cdb-4fd1-bbc1-15c089d827d9" (UID: "736322c2-4cdb-4fd1-bbc1-15c089d827d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.082084    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09665d83-c120-4c47-bc5f-45fa16149bb4-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.082145    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09665d83-c120-4c47-bc5f-45fa16149bb4-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.082162    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.082198    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.082216    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/903e1c2f-42e8-4702-8368-3831f717fd8e-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.082229    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09665d83-c120-4c47-bc5f-45fa16149bb4-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.082244    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg9ld\" (UniqueName: \"kubernetes.io/projected/09665d83-c120-4c47-bc5f-45fa16149bb4-kube-api-access-qg9ld\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.082286    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8d10219-d907-47b1-be66-70a4e9d40465-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.082301    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.082315    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.082328    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rfjt\" (UniqueName: \"kubernetes.io/projected/b8d10219-d907-47b1-be66-70a4e9d40465-kube-api-access-7rfjt\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.082359    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05a72fc0-2554-4f73-935e-ade8fb962a7c-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.082374    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.082386    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8d10219-d907-47b1-be66-70a4e9d40465-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.082401    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/903e1c2f-42e8-4702-8368-3831f717fd8e-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.082439    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/903e1c2f-42e8-4702-8368-3831f717fd8e-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.082454    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09665d83-c120-4c47-bc5f-45fa16149bb4-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.082467    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09665d83-c120-4c47-bc5f-45fa16149bb4-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.082480    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8d10219-d907-47b1-be66-70a4e9d40465-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.082493    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/903e1c2f-42e8-4702-8368-3831f717fd8e-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.082522    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.082538    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8d10219-d907-47b1-be66-70a4e9d40465-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.082551    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09665d83-c120-4c47-bc5f-45fa16149bb4-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.082564    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/903e1c2f-42e8-4702-8368-3831f717fd8e-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.082577    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/903e1c2f-42e8-4702-8368-3831f717fd8e-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.082608    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a72fc0-2554-4f73-935e-ade8fb962a7c-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.082622    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhq26\" (UniqueName: \"kubernetes.io/projected/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-kube-api-access-hhq26\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.082641    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/736322c2-4cdb-4fd1-bbc1-15c089d827d9-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.082690    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8d10219-d907-47b1-be66-70a4e9d40465-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.082706    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8d10219-d907-47b1-be66-70a4e9d40465-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.082719    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09665d83-c120-4c47-bc5f-45fa16149bb4-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.082732    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d10219-d907-47b1-be66-70a4e9d40465-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.082764    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/903e1c2f-42e8-4702-8368-3831f717fd8e-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.082783    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6jjt\" (UniqueName: \"kubernetes.io/projected/903e1c2f-42e8-4702-8368-3831f717fd8e-kube-api-access-t6jjt\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.082801    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.082845    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84cb8a9e-b4d1-4c1b-9338-2ea77abec886-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.083336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.082860    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d10219-d907-47b1-be66-70a4e9d40465-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.098875 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.088231    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd81aa48-0f88-44fb-8baa-db1deec4a2df-workload-socket" pod "fd81aa48-0f88-44fb-8baa-db1deec4a2df" (UID: "fd81aa48-0f88-44fb-8baa-db1deec4a2df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:39.098875 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.089275    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a72fc0-2554-4f73-935e-ade8fb962a7c-workload-socket" pod "05a72fc0-2554-4f73-935e-ade8fb962a7c" (UID: "05a72fc0-2554-4f73-935e-ade8fb962a7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:39.098875 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.089637    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/736322c2-4cdb-4fd1-bbc1-15c089d827d9-istiod-ca-cert" pod "736322c2-4cdb-4fd1-bbc1-15c089d827d9" (UID: "736322c2-4cdb-4fd1-bbc1-15c089d827d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:39.098875 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.094966    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd81aa48-0f88-44fb-8baa-db1deec4a2df-workload-certs" pod "fd81aa48-0f88-44fb-8baa-db1deec4a2df" (UID: "fd81aa48-0f88-44fb-8baa-db1deec4a2df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:39.098875 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.095188    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a72fc0-2554-4f73-935e-ade8fb962a7c-credential-socket" pod "05a72fc0-2554-4f73-935e-ade8fb962a7c" (UID: "05a72fc0-2554-4f73-935e-ade8fb962a7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:39.098875 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.095288    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd81aa48-0f88-44fb-8baa-db1deec4a2df-istio-data" pod "fd81aa48-0f88-44fb-8baa-db1deec4a2df" (UID: "fd81aa48-0f88-44fb-8baa-db1deec4a2df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:39.098875 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.095683    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd81aa48-0f88-44fb-8baa-db1deec4a2df-credential-socket" pod "fd81aa48-0f88-44fb-8baa-db1deec4a2df" (UID: "fd81aa48-0f88-44fb-8baa-db1deec4a2df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:39.149953 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.100259    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a72fc0-2554-4f73-935e-ade8fb962a7c-kube-api-access-q28mp" pod "05a72fc0-2554-4f73-935e-ade8fb962a7c" (UID: "05a72fc0-2554-4f73-935e-ade8fb962a7c"). InnerVolumeSpecName "kube-api-access-q28mp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:39.149953 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.100593    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736322c2-4cdb-4fd1-bbc1-15c089d827d9-workload-certs" pod "736322c2-4cdb-4fd1-bbc1-15c089d827d9" (UID: "736322c2-4cdb-4fd1-bbc1-15c089d827d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:39.149953 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.137367    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd81aa48-0f88-44fb-8baa-db1deec4a2df-istio-podinfo" pod "fd81aa48-0f88-44fb-8baa-db1deec4a2df" (UID: "fd81aa48-0f88-44fb-8baa-db1deec4a2df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:54:39.149953 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.138428    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a72fc0-2554-4f73-935e-ade8fb962a7c-istio-envoy" pod "05a72fc0-2554-4f73-935e-ade8fb962a7c" (UID: "05a72fc0-2554-4f73-935e-ade8fb962a7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:39.149953 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.138688    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736322c2-4cdb-4fd1-bbc1-15c089d827d9-istio-data" pod "736322c2-4cdb-4fd1-bbc1-15c089d827d9" (UID: "736322c2-4cdb-4fd1-bbc1-15c089d827d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:39.149953 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.138510    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736322c2-4cdb-4fd1-bbc1-15c089d827d9-istio-envoy" pod "736322c2-4cdb-4fd1-bbc1-15c089d827d9" (UID: "736322c2-4cdb-4fd1-bbc1-15c089d827d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:39.149953 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.138907    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736322c2-4cdb-4fd1-bbc1-15c089d827d9-credential-socket" pod "736322c2-4cdb-4fd1-bbc1-15c089d827d9" (UID: "736322c2-4cdb-4fd1-bbc1-15c089d827d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:39.149953 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.139212    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05a72fc0-2554-4f73-935e-ade8fb962a7c-istiod-ca-cert" pod "05a72fc0-2554-4f73-935e-ade8fb962a7c" (UID: "05a72fc0-2554-4f73-935e-ade8fb962a7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:39.149953 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.144640    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/736322c2-4cdb-4fd1-bbc1-15c089d827d9-istio-token" pod "736322c2-4cdb-4fd1-bbc1-15c089d827d9" (UID: "736322c2-4cdb-4fd1-bbc1-15c089d827d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:39.149953 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.147206    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd81aa48-0f88-44fb-8baa-db1deec4a2df-istiod-ca-cert" pod "fd81aa48-0f88-44fb-8baa-db1deec4a2df" (UID: "fd81aa48-0f88-44fb-8baa-db1deec4a2df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:39.165791 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.151528    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd81aa48-0f88-44fb-8baa-db1deec4a2df-istio-envoy" pod "fd81aa48-0f88-44fb-8baa-db1deec4a2df" (UID: "fd81aa48-0f88-44fb-8baa-db1deec4a2df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:39.165791 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.151777    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd81aa48-0f88-44fb-8baa-db1deec4a2df-istio-token" pod "fd81aa48-0f88-44fb-8baa-db1deec4a2df" (UID: "fd81aa48-0f88-44fb-8baa-db1deec4a2df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:39.165791 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.151899    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd81aa48-0f88-44fb-8baa-db1deec4a2df-kube-api-access-hjxq5" pod "fd81aa48-0f88-44fb-8baa-db1deec4a2df" (UID: "fd81aa48-0f88-44fb-8baa-db1deec4a2df"). InnerVolumeSpecName "kube-api-access-hjxq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:39.165791 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.152033    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/736322c2-4cdb-4fd1-bbc1-15c089d827d9-kube-api-access-k4hh9" pod "736322c2-4cdb-4fd1-bbc1-15c089d827d9" (UID: "736322c2-4cdb-4fd1-bbc1-15c089d827d9"). InnerVolumeSpecName "kube-api-access-k4hh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:39.165791 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.155583    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05a72fc0-2554-4f73-935e-ade8fb962a7c-istio-podinfo" pod "05a72fc0-2554-4f73-935e-ade8fb962a7c" (UID: "05a72fc0-2554-4f73-935e-ade8fb962a7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:54:39.165791 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.158467    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/736322c2-4cdb-4fd1-bbc1-15c089d827d9-istio-podinfo" pod "736322c2-4cdb-4fd1-bbc1-15c089d827d9" (UID: "736322c2-4cdb-4fd1-bbc1-15c089d827d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:54:39.165791 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.161275    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a72fc0-2554-4f73-935e-ade8fb962a7c-istio-token" pod "05a72fc0-2554-4f73-935e-ade8fb962a7c" (UID: "05a72fc0-2554-4f73-935e-ade8fb962a7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:39.169964 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.169938    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"]
Oct 30 20:54:39.170248 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.170218    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw" podUID="f2be68cc-8cd9-4e6e-893e-5d79ef5edea7" containerName="istio-proxy" containerID="cri-o://d5490695c2967cfa103ba2b6cb191fc835e8c43ab6e556a3b0b922c1b615b5c8" gracePeriod=30
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.183479    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5cdad370-2ff5-435c-bbfe-2ba61a325d47-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cdad370-2ff5-435c-bbfe-2ba61a325d47-istio-podinfo\") pod \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\" (UID: \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\") "
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.183526    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cdad370-2ff5-435c-bbfe-2ba61a325d47-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cdad370-2ff5-435c-bbfe-2ba61a325d47-istio-data\") pod \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\" (UID: \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\") "
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.183578    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-workload-certs\") pod \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\" (UID: \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\") "
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.183620    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-istio-token\" (UniqueName: \"kubernetes.io/projected/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-istio-token\") pod \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\" (UID: \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\") "
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.183666    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-kube-api-access-jdxfk\" (UniqueName: \"kubernetes.io/projected/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-kube-api-access-jdxfk\") pod \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\" (UID: \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\") "
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.183739    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-istio-podinfo\") pod \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\" (UID: \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\") "
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.183838    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cdad370-2ff5-435c-bbfe-2ba61a325d47-kube-api-access-vml6q\" (UniqueName: \"kubernetes.io/projected/5cdad370-2ff5-435c-bbfe-2ba61a325d47-kube-api-access-vml6q\") pod \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\" (UID: \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\") "
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.183897    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-workload-socket\") pod \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\" (UID: \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\") "
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.183959    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cdad370-2ff5-435c-bbfe-2ba61a325d47-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cdad370-2ff5-435c-bbfe-2ba61a325d47-workload-certs\") pod \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\" (UID: \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\") "
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.184001    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cdad370-2ff5-435c-bbfe-2ba61a325d47-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cdad370-2ff5-435c-bbfe-2ba61a325d47-credential-socket\") pod \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\" (UID: \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\") "
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.184037    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cdad370-2ff5-435c-bbfe-2ba61a325d47-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cdad370-2ff5-435c-bbfe-2ba61a325d47-istiod-ca-cert\") pod \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\" (UID: \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\") "
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.184069    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-istiod-ca-cert\") pod \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\" (UID: \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\") "
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.184155    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cdad370-2ff5-435c-bbfe-2ba61a325d47-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cdad370-2ff5-435c-bbfe-2ba61a325d47-workload-socket\") pod \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\" (UID: \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\") "
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.184238    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-credential-socket\") pod \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\" (UID: \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\") "
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.184284    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-istio-envoy\") pod \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\" (UID: \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\") "
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.184324    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cdad370-2ff5-435c-bbfe-2ba61a325d47-istio-token\" (UniqueName: \"kubernetes.io/projected/5cdad370-2ff5-435c-bbfe-2ba61a325d47-istio-token\") pod \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\" (UID: \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\") "
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.184390    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cdad370-2ff5-435c-bbfe-2ba61a325d47-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cdad370-2ff5-435c-bbfe-2ba61a325d47-istio-envoy\") pod \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\" (UID: \"5cdad370-2ff5-435c-bbfe-2ba61a325d47\") "
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.184468    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-istio-data\") pod \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\" (UID: \"723f1905-6d5a-45a7-a2bb-d5753a0fda9c\") "
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.185171    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05a72fc0-2554-4f73-935e-ade8fb962a7c-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.185192    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjxq5\" (UniqueName: \"kubernetes.io/projected/fd81aa48-0f88-44fb-8baa-db1deec4a2df-kube-api-access-hjxq5\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.185209    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/736322c2-4cdb-4fd1-bbc1-15c089d827d9-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.185224    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd81aa48-0f88-44fb-8baa-db1deec4a2df-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.185238    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/736322c2-4cdb-4fd1-bbc1-15c089d827d9-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.185255    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q28mp\" (UniqueName: \"kubernetes.io/projected/05a72fc0-2554-4f73-935e-ade8fb962a7c-kube-api-access-q28mp\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.185269    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd81aa48-0f88-44fb-8baa-db1deec4a2df-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.185293    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a72fc0-2554-4f73-935e-ade8fb962a7c-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.185308    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/736322c2-4cdb-4fd1-bbc1-15c089d827d9-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.185323    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd81aa48-0f88-44fb-8baa-db1deec4a2df-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.185337    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd81aa48-0f88-44fb-8baa-db1deec4a2df-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.185352    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd81aa48-0f88-44fb-8baa-db1deec4a2df-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.185366    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05a72fc0-2554-4f73-935e-ade8fb962a7c-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.185379    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd81aa48-0f88-44fb-8baa-db1deec4a2df-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.185392    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd81aa48-0f88-44fb-8baa-db1deec4a2df-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.185407    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/736322c2-4cdb-4fd1-bbc1-15c089d827d9-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.185422    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/736322c2-4cdb-4fd1-bbc1-15c089d827d9-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.185436    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4hh9\" (UniqueName: \"kubernetes.io/projected/736322c2-4cdb-4fd1-bbc1-15c089d827d9-kube-api-access-k4hh9\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.185449    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/736322c2-4cdb-4fd1-bbc1-15c089d827d9-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.185464    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/736322c2-4cdb-4fd1-bbc1-15c089d827d9-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.185480    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05a72fc0-2554-4f73-935e-ade8fb962a7c-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.185496    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a72fc0-2554-4f73-935e-ade8fb962a7c-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.185510    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd81aa48-0f88-44fb-8baa-db1deec4a2df-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.185753 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.185523    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05a72fc0-2554-4f73-935e-ade8fb962a7c-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.188399 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.188342    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-istio-data" pod "723f1905-6d5a-45a7-a2bb-d5753a0fda9c" (UID: "723f1905-6d5a-45a7-a2bb-d5753a0fda9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:39.188971 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.188740    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cdad370-2ff5-435c-bbfe-2ba61a325d47-workload-certs" pod "5cdad370-2ff5-435c-bbfe-2ba61a325d47" (UID: "5cdad370-2ff5-435c-bbfe-2ba61a325d47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:39.188971 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.188763    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cdad370-2ff5-435c-bbfe-2ba61a325d47-workload-socket" pod "5cdad370-2ff5-435c-bbfe-2ba61a325d47" (UID: "5cdad370-2ff5-435c-bbfe-2ba61a325d47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:39.189122 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.188922    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-istiod-ca-cert" pod "723f1905-6d5a-45a7-a2bb-d5753a0fda9c" (UID: "723f1905-6d5a-45a7-a2bb-d5753a0fda9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:39.189122 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.189009    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-credential-socket" pod "723f1905-6d5a-45a7-a2bb-d5753a0fda9c" (UID: "723f1905-6d5a-45a7-a2bb-d5753a0fda9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:39.189243 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.189212    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-workload-socket" pod "723f1905-6d5a-45a7-a2bb-d5753a0fda9c" (UID: "723f1905-6d5a-45a7-a2bb-d5753a0fda9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:39.189522 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.189290    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cdad370-2ff5-435c-bbfe-2ba61a325d47-istiod-ca-cert" pod "5cdad370-2ff5-435c-bbfe-2ba61a325d47" (UID: "5cdad370-2ff5-435c-bbfe-2ba61a325d47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:39.189522 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.189339    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cdad370-2ff5-435c-bbfe-2ba61a325d47-credential-socket" pod "5cdad370-2ff5-435c-bbfe-2ba61a325d47" (UID: "5cdad370-2ff5-435c-bbfe-2ba61a325d47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:39.189790 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.189710    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cdad370-2ff5-435c-bbfe-2ba61a325d47-istio-data" pod "5cdad370-2ff5-435c-bbfe-2ba61a325d47" (UID: "5cdad370-2ff5-435c-bbfe-2ba61a325d47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:39.189790 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.189774    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-workload-certs" pod "723f1905-6d5a-45a7-a2bb-d5753a0fda9c" (UID: "723f1905-6d5a-45a7-a2bb-d5753a0fda9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:39.190555 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.190525    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:39.191903 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.191864    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-istio-podinfo" pod "723f1905-6d5a-45a7-a2bb-d5753a0fda9c" (UID: "723f1905-6d5a-45a7-a2bb-d5753a0fda9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:54:39.194744 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.194722    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cdad370-2ff5-435c-bbfe-2ba61a325d47-istio-envoy" pod "5cdad370-2ff5-435c-bbfe-2ba61a325d47" (UID: "5cdad370-2ff5-435c-bbfe-2ba61a325d47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:39.196927 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.196902    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-istio-envoy" pod "723f1905-6d5a-45a7-a2bb-d5753a0fda9c" (UID: "723f1905-6d5a-45a7-a2bb-d5753a0fda9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:39.197023 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.196951    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cdad370-2ff5-435c-bbfe-2ba61a325d47-istio-token" pod "5cdad370-2ff5-435c-bbfe-2ba61a325d47" (UID: "5cdad370-2ff5-435c-bbfe-2ba61a325d47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:39.197023 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.196990    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cdad370-2ff5-435c-bbfe-2ba61a325d47-kube-api-access-vml6q" pod "5cdad370-2ff5-435c-bbfe-2ba61a325d47" (UID: "5cdad370-2ff5-435c-bbfe-2ba61a325d47"). InnerVolumeSpecName "kube-api-access-vml6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:39.197158 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.197052    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5cdad370-2ff5-435c-bbfe-2ba61a325d47-istio-podinfo" pod "5cdad370-2ff5-435c-bbfe-2ba61a325d47" (UID: "5cdad370-2ff5-435c-bbfe-2ba61a325d47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:54:39.198304 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.198281    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-kube-api-access-jdxfk" pod "723f1905-6d5a-45a7-a2bb-d5753a0fda9c" (UID: "723f1905-6d5a-45a7-a2bb-d5753a0fda9c"). InnerVolumeSpecName "kube-api-access-jdxfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:39.198929 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.198907    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-istio-token" pod "723f1905-6d5a-45a7-a2bb-d5753a0fda9c" (UID: "723f1905-6d5a-45a7-a2bb-d5753a0fda9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:39.237998 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.237969    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"
Oct 30 20:54:39.286763 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.286676    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa5037cf-aea2-403e-9865-8df55fe7e04f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5037cf-aea2-403e-9865-8df55fe7e04f-workload-socket\") pod \"aa5037cf-aea2-403e-9865-8df55fe7e04f\" (UID: \"aa5037cf-aea2-403e-9865-8df55fe7e04f\") "
Oct 30 20:54:39.286943 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.286879    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa5037cf-aea2-403e-9865-8df55fe7e04f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa5037cf-aea2-403e-9865-8df55fe7e04f-istio-data\") pod \"aa5037cf-aea2-403e-9865-8df55fe7e04f\" (UID: \"aa5037cf-aea2-403e-9865-8df55fe7e04f\") "
Oct 30 20:54:39.287010 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.286957    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa5037cf-aea2-403e-9865-8df55fe7e04f-workload-socket" pod "aa5037cf-aea2-403e-9865-8df55fe7e04f" (UID: "aa5037cf-aea2-403e-9865-8df55fe7e04f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:39.287010 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.286977    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa5037cf-aea2-403e-9865-8df55fe7e04f-istio-token\" (UniqueName: \"kubernetes.io/projected/aa5037cf-aea2-403e-9865-8df55fe7e04f-istio-token\") pod \"aa5037cf-aea2-403e-9865-8df55fe7e04f\" (UID: \"aa5037cf-aea2-403e-9865-8df55fe7e04f\") "
Oct 30 20:54:39.287134 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.287014    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa5037cf-aea2-403e-9865-8df55fe7e04f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa5037cf-aea2-403e-9865-8df55fe7e04f-istio-podinfo\") pod \"aa5037cf-aea2-403e-9865-8df55fe7e04f\" (UID: \"aa5037cf-aea2-403e-9865-8df55fe7e04f\") "
Oct 30 20:54:39.287134 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.287046    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa5037cf-aea2-403e-9865-8df55fe7e04f-kube-api-access-lczf9\" (UniqueName: \"kubernetes.io/projected/aa5037cf-aea2-403e-9865-8df55fe7e04f-kube-api-access-lczf9\") pod \"aa5037cf-aea2-403e-9865-8df55fe7e04f\" (UID: \"aa5037cf-aea2-403e-9865-8df55fe7e04f\") "
Oct 30 20:54:39.287233 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.287175    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa5037cf-aea2-403e-9865-8df55fe7e04f-istio-data" pod "aa5037cf-aea2-403e-9865-8df55fe7e04f" (UID: "aa5037cf-aea2-403e-9865-8df55fe7e04f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:39.287233 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.287181    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa5037cf-aea2-403e-9865-8df55fe7e04f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa5037cf-aea2-403e-9865-8df55fe7e04f-workload-certs\") pod \"aa5037cf-aea2-403e-9865-8df55fe7e04f\" (UID: \"aa5037cf-aea2-403e-9865-8df55fe7e04f\") "
Oct 30 20:54:39.287335 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.287241    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa5037cf-aea2-403e-9865-8df55fe7e04f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5037cf-aea2-403e-9865-8df55fe7e04f-credential-socket\") pod \"aa5037cf-aea2-403e-9865-8df55fe7e04f\" (UID: \"aa5037cf-aea2-403e-9865-8df55fe7e04f\") "
Oct 30 20:54:39.287388 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.287363    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa5037cf-aea2-403e-9865-8df55fe7e04f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa5037cf-aea2-403e-9865-8df55fe7e04f-istio-envoy\") pod \"aa5037cf-aea2-403e-9865-8df55fe7e04f\" (UID: \"aa5037cf-aea2-403e-9865-8df55fe7e04f\") "
Oct 30 20:54:39.287435 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.287404    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa5037cf-aea2-403e-9865-8df55fe7e04f-workload-certs" pod "aa5037cf-aea2-403e-9865-8df55fe7e04f" (UID: "aa5037cf-aea2-403e-9865-8df55fe7e04f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:39.287485 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.287469    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa5037cf-aea2-403e-9865-8df55fe7e04f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa5037cf-aea2-403e-9865-8df55fe7e04f-istiod-ca-cert\") pod \"aa5037cf-aea2-403e-9865-8df55fe7e04f\" (UID: \"aa5037cf-aea2-403e-9865-8df55fe7e04f\") "
Oct 30 20:54:39.287562 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.287536    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa5037cf-aea2-403e-9865-8df55fe7e04f-credential-socket" pod "aa5037cf-aea2-403e-9865-8df55fe7e04f" (UID: "aa5037cf-aea2-403e-9865-8df55fe7e04f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:39.288362 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.287934    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa5037cf-aea2-403e-9865-8df55fe7e04f-istiod-ca-cert" pod "aa5037cf-aea2-403e-9865-8df55fe7e04f" (UID: "aa5037cf-aea2-403e-9865-8df55fe7e04f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:39.288362 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.288272    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cdad370-2ff5-435c-bbfe-2ba61a325d47-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.288362 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.288293    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cdad370-2ff5-435c-bbfe-2ba61a325d47-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.288362 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.288308    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.288362 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.288322    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.288362 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.288336    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdxfk\" (UniqueName: \"kubernetes.io/projected/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-kube-api-access-jdxfk\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.288766 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.288380    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa5037cf-aea2-403e-9865-8df55fe7e04f-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.288766 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.288456    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5037cf-aea2-403e-9865-8df55fe7e04f-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.288766 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.288530    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.288766 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.288590    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vml6q\" (UniqueName: \"kubernetes.io/projected/5cdad370-2ff5-435c-bbfe-2ba61a325d47-kube-api-access-vml6q\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.288766 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.288606    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.288766 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.288620    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa5037cf-aea2-403e-9865-8df55fe7e04f-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.288766 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.288634    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cdad370-2ff5-435c-bbfe-2ba61a325d47-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.288766 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.288649    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cdad370-2ff5-435c-bbfe-2ba61a325d47-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.288766 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.288662    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cdad370-2ff5-435c-bbfe-2ba61a325d47-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.288766 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.288675    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.288766 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.288689    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cdad370-2ff5-435c-bbfe-2ba61a325d47-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.288766 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.288706    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.288766 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.288718    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.288766 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.288735    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cdad370-2ff5-435c-bbfe-2ba61a325d47-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.288766 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.288749    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa5037cf-aea2-403e-9865-8df55fe7e04f-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.288766 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.288762    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5037cf-aea2-403e-9865-8df55fe7e04f-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.288766 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.288777    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cdad370-2ff5-435c-bbfe-2ba61a325d47-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.289680 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.288791    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/723f1905-6d5a-45a7-a2bb-d5753a0fda9c-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.290352 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.290318    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa5037cf-aea2-403e-9865-8df55fe7e04f-istio-envoy" pod "aa5037cf-aea2-403e-9865-8df55fe7e04f" (UID: "aa5037cf-aea2-403e-9865-8df55fe7e04f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:39.290743 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.290714    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa5037cf-aea2-403e-9865-8df55fe7e04f-kube-api-access-lczf9" pod "aa5037cf-aea2-403e-9865-8df55fe7e04f" (UID: "aa5037cf-aea2-403e-9865-8df55fe7e04f"). InnerVolumeSpecName "kube-api-access-lczf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:39.290987 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.290961    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa5037cf-aea2-403e-9865-8df55fe7e04f-istio-podinfo" pod "aa5037cf-aea2-403e-9865-8df55fe7e04f" (UID: "aa5037cf-aea2-403e-9865-8df55fe7e04f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:54:39.291057 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.290997    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa5037cf-aea2-403e-9865-8df55fe7e04f-istio-token" pod "aa5037cf-aea2-403e-9865-8df55fe7e04f" (UID: "aa5037cf-aea2-403e-9865-8df55fe7e04f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:39.403844 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.393522    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa5037cf-aea2-403e-9865-8df55fe7e04f-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.403844 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.393561    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa5037cf-aea2-403e-9865-8df55fe7e04f-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.403844 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.393577    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa5037cf-aea2-403e-9865-8df55fe7e04f-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.403844 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.393592    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lczf9\" (UniqueName: \"kubernetes.io/projected/aa5037cf-aea2-403e-9865-8df55fe7e04f-kube-api-access-lczf9\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.682511 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.682477    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"
Oct 30 20:54:39.683456 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.683253    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g" event={"ID":"05a72fc0-2554-4f73-935e-ade8fb962a7c","Type":"ContainerDied","Data":"ada05eb7d0396ad1129ae1827bf3302b9163fc37ff9a7d791658cc4ed4eee994"}
Oct 30 20:54:39.683456 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.683332    5071 scope.go:122] "RemoveContainer" containerID="0d30aebc1f53767b2a8c9a23e00404dd3b977c5126fc9e26b5bc757dd3361f91"
Oct 30 20:54:39.686224 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.686047    5071 generic.go:356] "Generic (PLEG): container finished" podID="aa5037cf-aea2-403e-9865-8df55fe7e04f" containerID="4dab2753431dab7f9c6ebd9aae38bb413c72553660c8459a4ab0db453edc091a" exitCode=0
Oct 30 20:54:39.686224 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.686079    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl" event={"ID":"aa5037cf-aea2-403e-9865-8df55fe7e04f","Type":"ContainerDied","Data":"4dab2753431dab7f9c6ebd9aae38bb413c72553660c8459a4ab0db453edc091a"}
Oct 30 20:54:39.686224 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.686133    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl" event={"ID":"aa5037cf-aea2-403e-9865-8df55fe7e04f","Type":"ContainerDied","Data":"77c65115e69335137d7f32fdadccc40c770c29870f1ee1fa844d54b800e45eee"}
Oct 30 20:54:39.686224 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.686133    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"
Oct 30 20:54:39.688483 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.688026    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"
Oct 30 20:54:39.688483 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.688030    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s" event={"ID":"fd81aa48-0f88-44fb-8baa-db1deec4a2df","Type":"ContainerDied","Data":"5e34b39f6d58daccbf7cc7cede198d7d0df97d895d6b745fc48e3daeff716f0b"}
Oct 30 20:54:39.689949 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.689925    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87" event={"ID":"723f1905-6d5a-45a7-a2bb-d5753a0fda9c","Type":"ContainerDied","Data":"b320b360310298dbdfd63f7f280d9cd55bf22214be8a54bc3f411e759be94a60"}
Oct 30 20:54:39.690054 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.690018    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"
Oct 30 20:54:39.694453 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.694430    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj" event={"ID":"09665d83-c120-4c47-bc5f-45fa16149bb4","Type":"ContainerDied","Data":"da8f340962856d5869435ab11ff364cbb3530d1b7cb6dd6d70eb748ef8c2e8f5"}
Oct 30 20:54:39.694554 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.694465    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"
Oct 30 20:54:39.696246 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.696223    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92" event={"ID":"31b52815-7184-4c7b-a294-a2d9a7d0c099","Type":"ContainerDied","Data":"8aee0e0b76a26a8fce3b1015c7a77df189bf0e4d95fbd6088c4280eb931051c4"}
Oct 30 20:54:39.696347 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.696312    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"
Oct 30 20:54:39.698494 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.698470    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5" event={"ID":"b8d10219-d907-47b1-be66-70a4e9d40465","Type":"ContainerDied","Data":"b97a0a015edd4d1c30848d608bd4fc158626e390503072444557c99ed7dd66a2"}
Oct 30 20:54:39.698577 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.698565    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"
Oct 30 20:54:39.700622 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.700600    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t" event={"ID":"00b1175c-ea5d-49a8-b721-93fd709970c1","Type":"ContainerDied","Data":"0ce2edc5e2b46e3a69d4bb6a681ea8cf0d030ed5f99558d59f2623e3cb8175c8"}
Oct 30 20:54:39.700699 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.700692    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"
Oct 30 20:54:39.704844 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.702559    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"
Oct 30 20:54:39.704844 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.703427    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm" event={"ID":"903e1c2f-42e8-4702-8368-3831f717fd8e","Type":"ContainerDied","Data":"a8f8efcf70c2006488051e77bab41cd6e5104281b57c69faecd4abb6e95d5518"}
Oct 30 20:54:39.712543 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.712520    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"
Oct 30 20:54:39.712737 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.712716    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47" event={"ID":"736322c2-4cdb-4fd1-bbc1-15c089d827d9","Type":"ContainerDied","Data":"e49e39bcb7b2d095695e309bf78b3b4b7e4f353718671f50c9b9aad219b84955"}
Oct 30 20:54:39.728328 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.728239    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744" event={"ID":"84cb8a9e-b4d1-4c1b-9338-2ea77abec886","Type":"ContainerDied","Data":"4f9b850766e21965d77f426829fd998a8475f48968ced41a429163d9af36ec58"}
Oct 30 20:54:39.728545 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.728529    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"
Oct 30 20:54:39.733292 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.733274    5071 scope.go:122] "RemoveContainer" containerID="4dab2753431dab7f9c6ebd9aae38bb413c72553660c8459a4ab0db453edc091a"
Oct 30 20:54:39.742519 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.740982    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb" event={"ID":"5cdad370-2ff5-435c-bbfe-2ba61a325d47","Type":"ContainerDied","Data":"064e1ca3ff6e361bb67ba4354e3c41c2480b015c51792f410c6253b159b42d3d"}
Oct 30 20:54:39.742519 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.741184    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"
Oct 30 20:54:39.766997 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.766720    5071 scope.go:122] "RemoveContainer" containerID="4dab2753431dab7f9c6ebd9aae38bb413c72553660c8459a4ab0db453edc091a"
Oct 30 20:54:39.767211 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:54:39.767186    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dab2753431dab7f9c6ebd9aae38bb413c72553660c8459a4ab0db453edc091a\": container with ID starting with 4dab2753431dab7f9c6ebd9aae38bb413c72553660c8459a4ab0db453edc091a not found: ID does not exist" containerID="4dab2753431dab7f9c6ebd9aae38bb413c72553660c8459a4ab0db453edc091a"
Oct 30 20:54:39.767324 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.767218    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dab2753431dab7f9c6ebd9aae38bb413c72553660c8459a4ab0db453edc091a"} err="failed to get container status \"4dab2753431dab7f9c6ebd9aae38bb413c72553660c8459a4ab0db453edc091a\": rpc error: code = NotFound desc = could not find container \"4dab2753431dab7f9c6ebd9aae38bb413c72553660c8459a4ab0db453edc091a\": container with ID starting with 4dab2753431dab7f9c6ebd9aae38bb413c72553660c8459a4ab0db453edc091a not found: ID does not exist"
Oct 30 20:54:39.767324 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.767239    5071 scope.go:122] "RemoveContainer" containerID="753e257909c6531af48f1cafec5c39fb421f9e97fede074c88bf601b9071eb0f"
Oct 30 20:54:39.777969 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.777946    5071 scope.go:122] "RemoveContainer" containerID="b3689f02a0c436ed2add7966e749d0c6f7cc617b2bad25d8452f9e7bf29e240b"
Oct 30 20:54:39.788526 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.788500    5071 scope.go:122] "RemoveContainer" containerID="bf58317edd0ff572f75077ded61beb7eec136964bb1ece6e5c84a6a87cd40869"
Oct 30 20:54:39.799737 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.799714    5071 scope.go:122] "RemoveContainer" containerID="67664ec9a132ab24151dca469320d3494a4f9480c3c446b816cd116e47234e06"
Oct 30 20:54:39.812618 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.812596    5071 scope.go:122] "RemoveContainer" containerID="a900003585f1a457a8d53a69adc4c1f492f4d44f31ebb02cefb7fff6ac2c2183"
Oct 30 20:54:39.829085 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.829059    5071 scope.go:122] "RemoveContainer" containerID="15ede708ebc06744572625fcfd27ff799d5c51483946b49ccf1e095a9db1eab4"
Oct 30 20:54:39.841726 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.841701    5071 scope.go:122] "RemoveContainer" containerID="5ceeab6e98675cc7f87f3a1923620bc5c46f692c3b6077a80eeccfc61a22b189"
Oct 30 20:54:39.856004 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.855979    5071 scope.go:122] "RemoveContainer" containerID="2d13fba3913f9425b772b991272a49bbf4d1d0054558e2b5b5de3a4159d5fe4c"
Oct 30 20:54:39.866563 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.866541    5071 scope.go:122] "RemoveContainer" containerID="01f8afa45021133acda78d054117a722b14b334a8ed629371c812a4748bf5a3d"
Oct 30 20:54:39.877899 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:39.877881    5071 scope.go:122] "RemoveContainer" containerID="22a7dc2536154ac2157046ebc9b68adc172c85397eb9352cfc893e481db56048"
Oct 30 20:54:40.409362 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:40.409332    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"]
Oct 30 20:54:40.415388 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:40.415361    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-lfwz5"]
Oct 30 20:54:40.525051 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:40.525015    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"]
Oct 30 20:54:40.532866 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:40.532801    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-d4r5s"]
Oct 30 20:54:40.644002 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:40.643963    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"]
Oct 30 20:54:40.651615 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:40.651587    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-prccm"]
Oct 30 20:54:40.765553 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:40.765479    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"]
Oct 30 20:54:40.769617 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:40.769593    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-ktf92"]
Oct 30 20:54:40.881284 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:40.881249    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"]
Oct 30 20:54:40.888817 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:40.888786    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-v9744"]
Oct 30 20:54:41.004879 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:41.004802    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"]
Oct 30 20:54:41.008640 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:41.008615    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-4bfwj"]
Oct 30 20:54:41.120677 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:41.120645    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"]
Oct 30 20:54:41.124866 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:41.124842    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-sz6cb"]
Oct 30 20:54:41.236281 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:41.236250    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"]
Oct 30 20:54:41.243941 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:41.243906    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-g2p5g"]
Oct 30 20:54:41.357501 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:41.357470    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"]
Oct 30 20:54:41.361457 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:41.361428    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ee3fad-openshift-default-85fc9d9645-nwc47"]
Oct 30 20:54:41.472156 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:41.472073    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"]
Oct 30 20:54:41.479086 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:41.479061    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-bdm87"]
Oct 30 20:54:41.664169 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:41.664140    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"]
Oct 30 20:54:41.670626 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:41.670599    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f6ebf5-openshift-default-79567845f5-fstjl"]
Oct 30 20:54:41.782986 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:41.782902    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"]
Oct 30 20:54:41.786302 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:41.786277    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-395022-openshift-default-559878f94c-8g82t"]
Oct 30 20:54:42.204358 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:42.204322    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00b1175c-ea5d-49a8-b721-93fd709970c1" path="/var/lib/kubelet/pods/00b1175c-ea5d-49a8-b721-93fd709970c1/volumes"
Oct 30 20:54:42.204985 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:42.204963    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05a72fc0-2554-4f73-935e-ade8fb962a7c" path="/var/lib/kubelet/pods/05a72fc0-2554-4f73-935e-ade8fb962a7c/volumes"
Oct 30 20:54:42.205392 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:42.205380    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09665d83-c120-4c47-bc5f-45fa16149bb4" path="/var/lib/kubelet/pods/09665d83-c120-4c47-bc5f-45fa16149bb4/volumes"
Oct 30 20:54:42.206140 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:42.206125    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31b52815-7184-4c7b-a294-a2d9a7d0c099" path="/var/lib/kubelet/pods/31b52815-7184-4c7b-a294-a2d9a7d0c099/volumes"
Oct 30 20:54:42.206555 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:42.206542    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cdad370-2ff5-435c-bbfe-2ba61a325d47" path="/var/lib/kubelet/pods/5cdad370-2ff5-435c-bbfe-2ba61a325d47/volumes"
Oct 30 20:54:42.207001 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:42.206988    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="723f1905-6d5a-45a7-a2bb-d5753a0fda9c" path="/var/lib/kubelet/pods/723f1905-6d5a-45a7-a2bb-d5753a0fda9c/volumes"
Oct 30 20:54:42.207670 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:42.207657    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="736322c2-4cdb-4fd1-bbc1-15c089d827d9" path="/var/lib/kubelet/pods/736322c2-4cdb-4fd1-bbc1-15c089d827d9/volumes"
Oct 30 20:54:42.208115 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:42.208100    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84cb8a9e-b4d1-4c1b-9338-2ea77abec886" path="/var/lib/kubelet/pods/84cb8a9e-b4d1-4c1b-9338-2ea77abec886/volumes"
Oct 30 20:54:42.208500 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:42.208488    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="903e1c2f-42e8-4702-8368-3831f717fd8e" path="/var/lib/kubelet/pods/903e1c2f-42e8-4702-8368-3831f717fd8e/volumes"
Oct 30 20:54:42.209193 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:42.209179    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa5037cf-aea2-403e-9865-8df55fe7e04f" path="/var/lib/kubelet/pods/aa5037cf-aea2-403e-9865-8df55fe7e04f/volumes"
Oct 30 20:54:42.209617 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:42.209606    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8d10219-d907-47b1-be66-70a4e9d40465" path="/var/lib/kubelet/pods/b8d10219-d907-47b1-be66-70a4e9d40465/volumes"
Oct 30 20:54:42.210293 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:42.210279    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd81aa48-0f88-44fb-8baa-db1deec4a2df" path="/var/lib/kubelet/pods/fd81aa48-0f88-44fb-8baa-db1deec4a2df/volumes"
Oct 30 20:54:42.228273 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:42.228249    5071 patch_prober.go:29] interesting pod/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:54:42.228389 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:42.228297    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw" podUID="f2be68cc-8cd9-4e6e-893e-5d79ef5edea7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:54:43.395738 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.395700    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc"]
Oct 30 20:54:43.396436 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.396421    5071 cpu_manager.go:414] "containerMap: removing container" podUID="fd81aa48-0f88-44fb-8baa-db1deec4a2df" containerName="istio-proxy"
Oct 30 20:54:43.396482 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.396438    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd81aa48-0f88-44fb-8baa-db1deec4a2df" containerName="istio-proxy"
Oct 30 20:54:43.396482 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.396457    5071 cpu_manager.go:414] "containerMap: removing container" podUID="aa5037cf-aea2-403e-9865-8df55fe7e04f" containerName="istio-proxy"
Oct 30 20:54:43.396482 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.396462    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa5037cf-aea2-403e-9865-8df55fe7e04f" containerName="istio-proxy"
Oct 30 20:54:43.396482 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.396473    5071 cpu_manager.go:414] "containerMap: removing container" podUID="723f1905-6d5a-45a7-a2bb-d5753a0fda9c" containerName="istio-proxy"
Oct 30 20:54:43.396482 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.396479    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="723f1905-6d5a-45a7-a2bb-d5753a0fda9c" containerName="istio-proxy"
Oct 30 20:54:43.396625 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.396487    5071 cpu_manager.go:414] "containerMap: removing container" podUID="903e1c2f-42e8-4702-8368-3831f717fd8e" containerName="istio-proxy"
Oct 30 20:54:43.396625 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.396492    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="903e1c2f-42e8-4702-8368-3831f717fd8e" containerName="istio-proxy"
Oct 30 20:54:43.396625 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.396500    5071 cpu_manager.go:414] "containerMap: removing container" podUID="09665d83-c120-4c47-bc5f-45fa16149bb4" containerName="istio-proxy"
Oct 30 20:54:43.396625 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.396504    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09665d83-c120-4c47-bc5f-45fa16149bb4" containerName="istio-proxy"
Oct 30 20:54:43.396625 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.396511    5071 cpu_manager.go:414] "containerMap: removing container" podUID="31b52815-7184-4c7b-a294-a2d9a7d0c099" containerName="istio-proxy"
Oct 30 20:54:43.396625 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.396516    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31b52815-7184-4c7b-a294-a2d9a7d0c099" containerName="istio-proxy"
Oct 30 20:54:43.396625 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.396524    5071 cpu_manager.go:414] "containerMap: removing container" podUID="84cb8a9e-b4d1-4c1b-9338-2ea77abec886" containerName="istio-proxy"
Oct 30 20:54:43.396625 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.396529    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84cb8a9e-b4d1-4c1b-9338-2ea77abec886" containerName="istio-proxy"
Oct 30 20:54:43.396625 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.396536    5071 cpu_manager.go:414] "containerMap: removing container" podUID="5cdad370-2ff5-435c-bbfe-2ba61a325d47" containerName="istio-proxy"
Oct 30 20:54:43.396625 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.396540    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cdad370-2ff5-435c-bbfe-2ba61a325d47" containerName="istio-proxy"
Oct 30 20:54:43.396625 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.396555    5071 cpu_manager.go:414] "containerMap: removing container" podUID="00b1175c-ea5d-49a8-b721-93fd709970c1" containerName="istio-proxy"
Oct 30 20:54:43.396625 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.396559    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00b1175c-ea5d-49a8-b721-93fd709970c1" containerName="istio-proxy"
Oct 30 20:54:43.396625 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.396570    5071 cpu_manager.go:414] "containerMap: removing container" podUID="05a72fc0-2554-4f73-935e-ade8fb962a7c" containerName="istio-proxy"
Oct 30 20:54:43.396625 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.396574    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05a72fc0-2554-4f73-935e-ade8fb962a7c" containerName="istio-proxy"
Oct 30 20:54:43.396625 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.396580    5071 cpu_manager.go:414] "containerMap: removing container" podUID="b8d10219-d907-47b1-be66-70a4e9d40465" containerName="istio-proxy"
Oct 30 20:54:43.396625 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.396584    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8d10219-d907-47b1-be66-70a4e9d40465" containerName="istio-proxy"
Oct 30 20:54:43.396625 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.396589    5071 cpu_manager.go:414] "containerMap: removing container" podUID="736322c2-4cdb-4fd1-bbc1-15c089d827d9" containerName="istio-proxy"
Oct 30 20:54:43.396625 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.396594    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="736322c2-4cdb-4fd1-bbc1-15c089d827d9" containerName="istio-proxy"
Oct 30 20:54:43.397144 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.396706    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd81aa48-0f88-44fb-8baa-db1deec4a2df" containerName="istio-proxy"
Oct 30 20:54:43.397144 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.396716    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa5037cf-aea2-403e-9865-8df55fe7e04f" containerName="istio-proxy"
Oct 30 20:54:43.397144 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.396723    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="723f1905-6d5a-45a7-a2bb-d5753a0fda9c" containerName="istio-proxy"
Oct 30 20:54:43.397144 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.396730    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="903e1c2f-42e8-4702-8368-3831f717fd8e" containerName="istio-proxy"
Oct 30 20:54:43.397144 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.396736    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="09665d83-c120-4c47-bc5f-45fa16149bb4" containerName="istio-proxy"
Oct 30 20:54:43.397144 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.396741    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="31b52815-7184-4c7b-a294-a2d9a7d0c099" containerName="istio-proxy"
Oct 30 20:54:43.397144 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.396747    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="84cb8a9e-b4d1-4c1b-9338-2ea77abec886" containerName="istio-proxy"
Oct 30 20:54:43.397144 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.396753    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cdad370-2ff5-435c-bbfe-2ba61a325d47" containerName="istio-proxy"
Oct 30 20:54:43.397144 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.396761    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="00b1175c-ea5d-49a8-b721-93fd709970c1" containerName="istio-proxy"
Oct 30 20:54:43.397144 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.396766    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="05a72fc0-2554-4f73-935e-ade8fb962a7c" containerName="istio-proxy"
Oct 30 20:54:43.397144 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.396772    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8d10219-d907-47b1-be66-70a4e9d40465" containerName="istio-proxy"
Oct 30 20:54:43.397144 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.396777    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="736322c2-4cdb-4fd1-bbc1-15c089d827d9" containerName="istio-proxy"
Oct 30 20:54:43.401962 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.401945    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc"
Oct 30 20:54:43.461702 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.461664    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc"]
Oct 30 20:54:43.481498 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.481467    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a458b472-4aae-4dcc-87c9-cadd0a317061-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc\" (UID: \"a458b472-4aae-4dcc-87c9-cadd0a317061\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc"
Oct 30 20:54:43.481650 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.481577    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wkb6\" (UniqueName: \"kubernetes.io/projected/a458b472-4aae-4dcc-87c9-cadd0a317061-kube-api-access-5wkb6\") pod \"maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc\" (UID: \"a458b472-4aae-4dcc-87c9-cadd0a317061\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc"
Oct 30 20:54:43.481650 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.481602    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a458b472-4aae-4dcc-87c9-cadd0a317061-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc\" (UID: \"a458b472-4aae-4dcc-87c9-cadd0a317061\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc"
Oct 30 20:54:43.481650 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.481634    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a458b472-4aae-4dcc-87c9-cadd0a317061-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc\" (UID: \"a458b472-4aae-4dcc-87c9-cadd0a317061\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc"
Oct 30 20:54:43.504502 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.504466    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-aaf669-serving-cert\""
Oct 30 20:54:43.583764 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.583724    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a458b472-4aae-4dcc-87c9-cadd0a317061-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc\" (UID: \"a458b472-4aae-4dcc-87c9-cadd0a317061\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc"
Oct 30 20:54:43.583949 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.583811    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wkb6\" (UniqueName: \"kubernetes.io/projected/a458b472-4aae-4dcc-87c9-cadd0a317061-kube-api-access-5wkb6\") pod \"maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc\" (UID: \"a458b472-4aae-4dcc-87c9-cadd0a317061\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc"
Oct 30 20:54:43.583949 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.583860    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a458b472-4aae-4dcc-87c9-cadd0a317061-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc\" (UID: \"a458b472-4aae-4dcc-87c9-cadd0a317061\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc"
Oct 30 20:54:43.583949 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.583892    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a458b472-4aae-4dcc-87c9-cadd0a317061-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc\" (UID: \"a458b472-4aae-4dcc-87c9-cadd0a317061\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc"
Oct 30 20:54:43.583949 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:54:43.583919    5071 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-aaf669-serving-cert: secret "maas-api-e2e-shared-b-w2-aaf669-serving-cert" not found
Oct 30 20:54:43.584145 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:54:43.583997    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a458b472-4aae-4dcc-87c9-cadd0a317061-maas-api-tls podName:a458b472-4aae-4dcc-87c9-cadd0a317061 nodeName:}" failed. No retries permitted until 2026-10-30 20:54:44.083976218 +0000 UTC m=+1696.000997742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a458b472-4aae-4dcc-87c9-cadd0a317061-maas-api-tls") pod "maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc" (UID: "a458b472-4aae-4dcc-87c9-cadd0a317061") : secret "maas-api-e2e-shared-b-w2-aaf669-serving-cert" not found
Oct 30 20:54:43.584729 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.584704    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a458b472-4aae-4dcc-87c9-cadd0a317061-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc\" (UID: \"a458b472-4aae-4dcc-87c9-cadd0a317061\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc"
Oct 30 20:54:43.586653 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.586621    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a458b472-4aae-4dcc-87c9-cadd0a317061-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc\" (UID: \"a458b472-4aae-4dcc-87c9-cadd0a317061\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc"
Oct 30 20:54:43.612780 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.612746    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:43.688901 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.685135    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wkb6\" (UniqueName: \"kubernetes.io/projected/a458b472-4aae-4dcc-87c9-cadd0a317061-kube-api-access-5wkb6\") pod \"maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc\" (UID: \"a458b472-4aae-4dcc-87c9-cadd0a317061\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc"
Oct 30 20:54:43.795074 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.794970    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:43.969671 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:43.969587    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:44.103021 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:44.102981    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a458b472-4aae-4dcc-87c9-cadd0a317061-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc\" (UID: \"a458b472-4aae-4dcc-87c9-cadd0a317061\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc"
Oct 30 20:54:44.105669 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:44.105643    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a458b472-4aae-4dcc-87c9-cadd0a317061-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc\" (UID: \"a458b472-4aae-4dcc-87c9-cadd0a317061\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc"
Oct 30 20:54:44.317580 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:44.317544    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc"
Oct 30 20:54:44.711504 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:44.711474    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc"]
Oct 30 20:54:44.712907 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:54:44.712878    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda458b472_4aae_4dcc_87c9_cadd0a317061.slice/crio-bd6fe7760ff1f19aa4791edeb2bd39f202da4c514a7e476db4876fb5902b59c7 WatchSource:0}: Error finding container bd6fe7760ff1f19aa4791edeb2bd39f202da4c514a7e476db4876fb5902b59c7: Status 404 returned error can't find the container with id bd6fe7760ff1f19aa4791edeb2bd39f202da4c514a7e476db4876fb5902b59c7
Oct 30 20:54:44.806517 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:44.806475    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc" event={"ID":"a458b472-4aae-4dcc-87c9-cadd0a317061","Type":"ContainerStarted","Data":"bd6fe7760ff1f19aa4791edeb2bd39f202da4c514a7e476db4876fb5902b59c7"}
Oct 30 20:54:44.809490 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:44.809463    5071 generic.go:356] "Generic (PLEG): container finished" podID="f2be68cc-8cd9-4e6e-893e-5d79ef5edea7" containerID="d5490695c2967cfa103ba2b6cb191fc835e8c43ab6e556a3b0b922c1b615b5c8" exitCode=0
Oct 30 20:54:44.809649 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:44.809539    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw" event={"ID":"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7","Type":"ContainerDied","Data":"d5490695c2967cfa103ba2b6cb191fc835e8c43ab6e556a3b0b922c1b615b5c8"}
Oct 30 20:54:44.809649 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:44.809613    5071 scope.go:122] "RemoveContainer" containerID="7c0db49e869689b12951479f4020a9c030ee6c81417955579decec1e993e1c60"
Oct 30 20:54:44.878093 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:44.878067    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:54:45.016107 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:45.016018    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-kube-api-access-q9xtm\" (UniqueName: \"kubernetes.io/projected/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-kube-api-access-q9xtm\") pod \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\" (UID: \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\") "
Oct 30 20:54:45.016107 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:45.016088    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-credential-socket\") pod \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\" (UID: \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\") "
Oct 30 20:54:45.016335 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:45.016124    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-istio-envoy\") pod \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\" (UID: \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\") "
Oct 30 20:54:45.016389 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:45.016354    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-istio-data\") pod \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\" (UID: \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\") "
Oct 30 20:54:45.016442 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:45.016399    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-workload-certs\") pod \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\" (UID: \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\") "
Oct 30 20:54:45.016442 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:45.016431    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-istio-token\" (UniqueName: \"kubernetes.io/projected/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-istio-token\") pod \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\" (UID: \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\") "
Oct 30 20:54:45.016548 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:45.016467    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-istio-podinfo\") pod \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\" (UID: \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\") "
Oct 30 20:54:45.016548 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:45.016523    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-istiod-ca-cert\") pod \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\" (UID: \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\") "
Oct 30 20:54:45.016645 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:45.016571    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-workload-socket\") pod \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\" (UID: \"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7\") "
Oct 30 20:54:45.017334 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:45.017306    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-istio-data" pod "f2be68cc-8cd9-4e6e-893e-5d79ef5edea7" (UID: "f2be68cc-8cd9-4e6e-893e-5d79ef5edea7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:45.017798 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:45.017466    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-istiod-ca-cert" pod "f2be68cc-8cd9-4e6e-893e-5d79ef5edea7" (UID: "f2be68cc-8cd9-4e6e-893e-5d79ef5edea7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:45.017798 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:45.017642    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-credential-socket" pod "f2be68cc-8cd9-4e6e-893e-5d79ef5edea7" (UID: "f2be68cc-8cd9-4e6e-893e-5d79ef5edea7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:45.017798 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:45.017766    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-workload-certs" pod "f2be68cc-8cd9-4e6e-893e-5d79ef5edea7" (UID: "f2be68cc-8cd9-4e6e-893e-5d79ef5edea7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:45.017798 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:45.017780    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-workload-socket" pod "f2be68cc-8cd9-4e6e-893e-5d79ef5edea7" (UID: "f2be68cc-8cd9-4e6e-893e-5d79ef5edea7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:45.020164 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:45.020137    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-istio-podinfo" pod "f2be68cc-8cd9-4e6e-893e-5d79ef5edea7" (UID: "f2be68cc-8cd9-4e6e-893e-5d79ef5edea7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:54:45.023749 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:45.023655    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-istio-token" pod "f2be68cc-8cd9-4e6e-893e-5d79ef5edea7" (UID: "f2be68cc-8cd9-4e6e-893e-5d79ef5edea7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:45.023749 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:45.023689    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-istio-envoy" pod "f2be68cc-8cd9-4e6e-893e-5d79ef5edea7" (UID: "f2be68cc-8cd9-4e6e-893e-5d79ef5edea7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:54:45.023749 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:45.023694    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-kube-api-access-q9xtm" pod "f2be68cc-8cd9-4e6e-893e-5d79ef5edea7" (UID: "f2be68cc-8cd9-4e6e-893e-5d79ef5edea7"). InnerVolumeSpecName "kube-api-access-q9xtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:45.119114 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:45.119079    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9xtm\" (UniqueName: \"kubernetes.io/projected/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-kube-api-access-q9xtm\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:45.119114 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:45.119111    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:45.119309 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:45.119129    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:45.119309 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:45.119138    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:45.119309 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:45.119146    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:45.119309 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:45.119155    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:45.119309 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:45.119163    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:45.119309 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:45.119171    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:45.119309 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:45.119179    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:45.302152 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:45.302124    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d9b3f6-86cfc8c66c-l48wg"
Oct 30 20:54:45.302323 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:45.302198    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d9b3f6-845694f564-8jdjx"
Oct 30 20:54:45.819920 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:45.819832    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"
Oct 30 20:54:45.819920 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:45.819856    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw" event={"ID":"f2be68cc-8cd9-4e6e-893e-5d79ef5edea7","Type":"ContainerDied","Data":"07caa6c18f91cd8f5fa72eca89be1cc302cdd9f546bb329432f3426735b5b5fa"}
Oct 30 20:54:45.819920 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:45.819913    5071 scope.go:122] "RemoveContainer" containerID="d5490695c2967cfa103ba2b6cb191fc835e8c43ab6e556a3b0b922c1b615b5c8"
Oct 30 20:54:45.821809 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:45.821786    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc" event={"ID":"a458b472-4aae-4dcc-87c9-cadd0a317061","Type":"ContainerStarted","Data":"7215e93e640b303c5a0ea469fa080c2216dd684f50b8bb5cacbb064913f49a3f"}
Oct 30 20:54:45.821991 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:45.821976    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc"
Oct 30 20:54:45.997654 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:45.997597    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc" podStartSLOduration=2.146626191 podStartE2EDuration="2.997581289s" totalImagesPullingTime="850.955098ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:45.99512463 +0000 UTC m=+1697.912146166" watchObservedRunningTime="2026-10-30 20:54:45.997581289 +0000 UTC m=+1697.914602825"
Oct 30 20:54:46.180933 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:46.178669    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"]
Oct 30 20:54:46.185855 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:46.185808    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ad0ff4fc-openshift-default-574b56dbd5-cwjhw"]
Oct 30 20:54:46.204701 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:46.204672    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2be68cc-8cd9-4e6e-893e-5d79ef5edea7" path="/var/lib/kubelet/pods/f2be68cc-8cd9-4e6e-893e-5d79ef5edea7/volumes"
Oct 30 20:54:46.730218 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:54:46.730177    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:54:48.901725 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:48.901686    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:49.298031 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:49.297993    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:51.838072 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:51.838044    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc"
Oct 30 20:54:54.126957 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:54.126620    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:56.075161 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:56.075128    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-86d620-74c5fd4b96-2zkm5"]
Oct 30 20:54:56.197950 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:56.192551    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-86d620-fb8d7795f-mjmqg"]
Oct 30 20:54:56.476076 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:56.475965    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:56.657761 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:56.657727    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:58.755705 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:58.755674    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7"]
Oct 30 20:54:58.756512 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:58.756494    5071 cpu_manager.go:414] "containerMap: removing container" podUID="f2be68cc-8cd9-4e6e-893e-5d79ef5edea7" containerName="istio-proxy"
Oct 30 20:54:58.756574 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:58.756514    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2be68cc-8cd9-4e6e-893e-5d79ef5edea7" containerName="istio-proxy"
Oct 30 20:54:58.756574 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:58.756537    5071 cpu_manager.go:414] "containerMap: removing container" podUID="f2be68cc-8cd9-4e6e-893e-5d79ef5edea7" containerName="istio-proxy"
Oct 30 20:54:58.756574 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:58.756544    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2be68cc-8cd9-4e6e-893e-5d79ef5edea7" containerName="istio-proxy"
Oct 30 20:54:58.756686 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:58.756655    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2be68cc-8cd9-4e6e-893e-5d79ef5edea7" containerName="istio-proxy"
Oct 30 20:54:58.756686 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:58.756668    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2be68cc-8cd9-4e6e-893e-5d79ef5edea7" containerName="istio-proxy"
Oct 30 20:54:58.762055 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:58.762037    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7"
Oct 30 20:54:58.818363 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:58.818333    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-3d7c11-serving-cert\""
Oct 30 20:54:58.869551 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:58.869512    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a31fc448-1791-4511-9b2f-c90e94928e77-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7\" (UID: \"a31fc448-1791-4511-9b2f-c90e94928e77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7"
Oct 30 20:54:58.869880 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:58.869576    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a31fc448-1791-4511-9b2f-c90e94928e77-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7\" (UID: \"a31fc448-1791-4511-9b2f-c90e94928e77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7"
Oct 30 20:54:58.869880 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:58.869696    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a31fc448-1791-4511-9b2f-c90e94928e77-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7\" (UID: \"a31fc448-1791-4511-9b2f-c90e94928e77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7"
Oct 30 20:54:58.869880 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:58.869726    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsfvb\" (UniqueName: \"kubernetes.io/projected/a31fc448-1791-4511-9b2f-c90e94928e77-kube-api-access-fsfvb\") pod \"maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7\" (UID: \"a31fc448-1791-4511-9b2f-c90e94928e77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7"
Oct 30 20:54:58.874613 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:58.874586    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7"]
Oct 30 20:54:58.971023 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:58.970957    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a31fc448-1791-4511-9b2f-c90e94928e77-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7\" (UID: \"a31fc448-1791-4511-9b2f-c90e94928e77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7"
Oct 30 20:54:58.971023 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:58.971009    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a31fc448-1791-4511-9b2f-c90e94928e77-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7\" (UID: \"a31fc448-1791-4511-9b2f-c90e94928e77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7"
Oct 30 20:54:58.971283 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:58.971126    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a31fc448-1791-4511-9b2f-c90e94928e77-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7\" (UID: \"a31fc448-1791-4511-9b2f-c90e94928e77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7"
Oct 30 20:54:58.971283 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:54:58.971142    5071 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3d7c11-serving-cert: secret "maas-api-e2e-shared-a-w6-3d7c11-serving-cert" not found
Oct 30 20:54:58.971283 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:58.971154    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsfvb\" (UniqueName: \"kubernetes.io/projected/a31fc448-1791-4511-9b2f-c90e94928e77-kube-api-access-fsfvb\") pod \"maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7\" (UID: \"a31fc448-1791-4511-9b2f-c90e94928e77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7"
Oct 30 20:54:58.971283 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:54:58.971217    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a31fc448-1791-4511-9b2f-c90e94928e77-maas-api-tls podName:a31fc448-1791-4511-9b2f-c90e94928e77 nodeName:}" failed. No retries permitted until 2026-10-30 20:54:59.471193232 +0000 UTC m=+1711.388214759 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a31fc448-1791-4511-9b2f-c90e94928e77-maas-api-tls") pod "maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7" (UID: "a31fc448-1791-4511-9b2f-c90e94928e77") : secret "maas-api-e2e-shared-a-w6-3d7c11-serving-cert" not found
Oct 30 20:54:58.972627 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:58.972601    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a31fc448-1791-4511-9b2f-c90e94928e77-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7\" (UID: \"a31fc448-1791-4511-9b2f-c90e94928e77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7"
Oct 30 20:54:58.972900 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:58.972869    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:58.974314 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:58.974292    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a31fc448-1791-4511-9b2f-c90e94928e77-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7\" (UID: \"a31fc448-1791-4511-9b2f-c90e94928e77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7"
Oct 30 20:54:59.054575 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:59.054531    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsfvb\" (UniqueName: \"kubernetes.io/projected/a31fc448-1791-4511-9b2f-c90e94928e77-kube-api-access-fsfvb\") pod \"maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7\" (UID: \"a31fc448-1791-4511-9b2f-c90e94928e77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7"
Oct 30 20:54:59.331748 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:59.331629    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:59.478630 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:59.478593    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a31fc448-1791-4511-9b2f-c90e94928e77-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7\" (UID: \"a31fc448-1791-4511-9b2f-c90e94928e77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7"
Oct 30 20:54:59.481523 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:59.481496    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a31fc448-1791-4511-9b2f-c90e94928e77-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7\" (UID: \"a31fc448-1791-4511-9b2f-c90e94928e77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7"
Oct 30 20:54:59.507097 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:59.507061    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:54:59.676792 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:54:59.676717    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7"
Oct 30 20:55:00.063108 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:00.063078    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7"]
Oct 30 20:55:00.064022 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:55:00.063986    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda31fc448_1791_4511_9b2f_c90e94928e77.slice/crio-9831d0c2e9e6c9864d175204ac19f87b3d3c79d98bbf7c6c54490d512a730434 WatchSource:0}: Error finding container 9831d0c2e9e6c9864d175204ac19f87b3d3c79d98bbf7c6c54490d512a730434: Status 404 returned error can't find the container with id 9831d0c2e9e6c9864d175204ac19f87b3d3c79d98bbf7c6c54490d512a730434
Oct 30 20:55:00.971164 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:00.971131    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7" event={"ID":"a31fc448-1791-4511-9b2f-c90e94928e77","Type":"ContainerStarted","Data":"5e249dccc783be6471b315abd64c65b9aee90c06cc103767fcee4159ba0c7f04"}
Oct 30 20:55:00.971361 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:00.971197    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7" event={"ID":"a31fc448-1791-4511-9b2f-c90e94928e77","Type":"ContainerStarted","Data":"9831d0c2e9e6c9864d175204ac19f87b3d3c79d98bbf7c6c54490d512a730434"}
Oct 30 20:55:00.971361 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:00.971343    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7"
Oct 30 20:55:01.075854 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:01.075770    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-86d620-74c5fd4b96-2zkm5" podUID="514c415b-492e-4092-9889-c2835f58b761" containerName="payload-processing" containerID="cri-o://1df1e4d8ec1594c9a65542c5944f756af17a466de11a134503ed67b3f99765a4" gracePeriod=25
Oct 30 20:55:01.138284 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:01.138185    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7" podStartSLOduration=2.362255589 podStartE2EDuration="3.138166665s" totalImagesPullingTime="775.911076ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:01.134488998 +0000 UTC m=+1713.051510534" watchObservedRunningTime="2026-10-30 20:55:01.138166665 +0000 UTC m=+1713.055188207"
Oct 30 20:55:01.193402 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:01.193364    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-86d620-fb8d7795f-mjmqg" podUID="904e938e-ad13-4460-9ab6-b1ce6ed3c1d6" containerName="payload-pre-processing" containerID="cri-o://4c9dfe7e186d197b98d166719328a4df91bcb0319732c03a60124dcc9c986061" gracePeriod=25
Oct 30 20:55:04.203431 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:04.203396    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:06.512752 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:06.512713    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-86d620-74c5fd4b96-2zkm5" podUID="514c415b-492e-4092-9889-c2835f58b761" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:55:06.513187 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:06.512719    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-86d620-fb8d7795f-mjmqg" podUID="904e938e-ad13-4460-9ab6-b1ce6ed3c1d6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:55:07.002897 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:07.002870    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7"
Oct 30 20:55:09.122240 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:09.122196    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:14.395068 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:14.395027    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:16.512677 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:16.512633    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-86d620-fb8d7795f-mjmqg" podUID="904e938e-ad13-4460-9ab6-b1ce6ed3c1d6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:55:16.512677 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:16.512646    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-86d620-74c5fd4b96-2zkm5" podUID="514c415b-492e-4092-9889-c2835f58b761" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:55:16.729855 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:55:16.729798    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:55:19.014691 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:19.014658    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd"]
Oct 30 20:55:19.020244 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:19.020222    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd"
Oct 30 20:55:19.030768 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:19.030740    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1203632-126c-4620-8740-6f07120ad6cd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd\" (UID: \"b1203632-126c-4620-8740-6f07120ad6cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd"
Oct 30 20:55:19.030918 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:19.030800    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1203632-126c-4620-8740-6f07120ad6cd-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd\" (UID: \"b1203632-126c-4620-8740-6f07120ad6cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd"
Oct 30 20:55:19.030977 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:19.030957    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mxxm\" (UniqueName: \"kubernetes.io/projected/b1203632-126c-4620-8740-6f07120ad6cd-kube-api-access-7mxxm\") pod \"maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd\" (UID: \"b1203632-126c-4620-8740-6f07120ad6cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd"
Oct 30 20:55:19.031058 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:19.031042    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1203632-126c-4620-8740-6f07120ad6cd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd\" (UID: \"b1203632-126c-4620-8740-6f07120ad6cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd"
Oct 30 20:55:19.078278 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:19.078247    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd"]
Oct 30 20:55:19.121134 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:19.121101    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-15199a-serving-cert\""
Oct 30 20:55:19.132371 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:19.132343    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1203632-126c-4620-8740-6f07120ad6cd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd\" (UID: \"b1203632-126c-4620-8740-6f07120ad6cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd"
Oct 30 20:55:19.132546 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:19.132409    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1203632-126c-4620-8740-6f07120ad6cd-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd\" (UID: \"b1203632-126c-4620-8740-6f07120ad6cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd"
Oct 30 20:55:19.132546 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:19.132507    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mxxm\" (UniqueName: \"kubernetes.io/projected/b1203632-126c-4620-8740-6f07120ad6cd-kube-api-access-7mxxm\") pod \"maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd\" (UID: \"b1203632-126c-4620-8740-6f07120ad6cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd"
Oct 30 20:55:19.132546 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:55:19.132522    5071 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-15199a-serving-cert: secret "maas-api-e2e-shared-b-w5-15199a-serving-cert" not found
Oct 30 20:55:19.132756 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:19.132576    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1203632-126c-4620-8740-6f07120ad6cd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd\" (UID: \"b1203632-126c-4620-8740-6f07120ad6cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd"
Oct 30 20:55:19.132756 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:55:19.132592    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1203632-126c-4620-8740-6f07120ad6cd-maas-api-tls podName:b1203632-126c-4620-8740-6f07120ad6cd nodeName:}" failed. No retries permitted until 2026-10-30 20:55:19.632570999 +0000 UTC m=+1731.549592516 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b1203632-126c-4620-8740-6f07120ad6cd-maas-api-tls") pod "maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd" (UID: "b1203632-126c-4620-8740-6f07120ad6cd") : secret "maas-api-e2e-shared-b-w5-15199a-serving-cert" not found
Oct 30 20:55:19.133377 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:19.133328    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1203632-126c-4620-8740-6f07120ad6cd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd\" (UID: \"b1203632-126c-4620-8740-6f07120ad6cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd"
Oct 30 20:55:19.135460 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:19.135429    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1203632-126c-4620-8740-6f07120ad6cd-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd\" (UID: \"b1203632-126c-4620-8740-6f07120ad6cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd"
Oct 30 20:55:19.200641 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:19.200607    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mxxm\" (UniqueName: \"kubernetes.io/projected/b1203632-126c-4620-8740-6f07120ad6cd-kube-api-access-7mxxm\") pod \"maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd\" (UID: \"b1203632-126c-4620-8740-6f07120ad6cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd"
Oct 30 20:55:19.246034 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:19.245999    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:19.438482 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:19.438442    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:19.609439 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:19.609401    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:19.645884 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:19.645852    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1203632-126c-4620-8740-6f07120ad6cd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd\" (UID: \"b1203632-126c-4620-8740-6f07120ad6cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd"
Oct 30 20:55:19.648736 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:19.648701    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1203632-126c-4620-8740-6f07120ad6cd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd\" (UID: \"b1203632-126c-4620-8740-6f07120ad6cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd"
Oct 30 20:55:19.869710 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:19.867885    5071 patch_prober.go:29] interesting pod/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": read tcp 10.132.0.2:57960->10.132.0.159:15021: read: connection reset by peer" start-of-body=
Oct 30 20:55:19.869710 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:19.867937    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt" podUID="d204447d-7147-415b-abf8-853f68b51fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": read tcp 10.132.0.2:57960->10.132.0.159:15021: read: connection reset by peer"
Oct 30 20:55:19.940492 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:19.940461    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd"
Oct 30 20:55:20.147573 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:20.147481    5071 generic.go:356] "Generic (PLEG): container finished" podID="d204447d-7147-415b-abf8-853f68b51fec" containerID="7da7aee651df96f9cb6a880e8f7d572a730af37552f32977d48c2c16a6c2edaa" exitCode=0
Oct 30 20:55:20.147573 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:20.147561    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt" event={"ID":"d204447d-7147-415b-abf8-853f68b51fec","Type":"ContainerDied","Data":"7da7aee651df96f9cb6a880e8f7d572a730af37552f32977d48c2c16a6c2edaa"}
Oct 30 20:55:20.148150 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:20.148071    5071 scope.go:122] "RemoveContainer" containerID="7da7aee651df96f9cb6a880e8f7d572a730af37552f32977d48c2c16a6c2edaa"
Oct 30 20:55:20.149857 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:20.149814    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:55:20.149959 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:20.149896    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:55:20.149959 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:20.149932    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:55:20.328039 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:20.327989    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd"]
Oct 30 20:55:21.173663 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:21.173630    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt" event={"ID":"d204447d-7147-415b-abf8-853f68b51fec","Type":"ContainerStarted","Data":"3ac4564610203faa0abbfc3295a6b9971d34f0f348388302667edb1845e97926"}
Oct 30 20:55:21.175188 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:21.175149    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd" event={"ID":"b1203632-126c-4620-8740-6f07120ad6cd","Type":"ContainerStarted","Data":"fc52771cc920103370c4601c8fe2a947c66245215f72a37294054e904491f982"}
Oct 30 20:55:21.520426 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:21.520395    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:55:21.520648 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:21.520518    5071 patch_prober.go:29] interesting pod/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 20:55:21.520648 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:21.520563    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt" podUID="d204447d-7147-415b-abf8-853f68b51fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 20:55:21.627149 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:21.627124    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-86d620-74c5fd4b96-2zkm5"
Oct 30 20:55:21.682747 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:21.682654    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/514c415b-492e-4092-9889-c2835f58b761-kube-api-access-dzm8s\" (UniqueName: \"kubernetes.io/projected/514c415b-492e-4092-9889-c2835f58b761-kube-api-access-dzm8s\") pod \"514c415b-492e-4092-9889-c2835f58b761\" (UID: \"514c415b-492e-4092-9889-c2835f58b761\") "
Oct 30 20:55:21.682980 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:21.682944    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/514c415b-492e-4092-9889-c2835f58b761-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/514c415b-492e-4092-9889-c2835f58b761-plugins-config-volume\") pod \"514c415b-492e-4092-9889-c2835f58b761\" (UID: \"514c415b-492e-4092-9889-c2835f58b761\") "
Oct 30 20:55:21.683560 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:21.683529    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/514c415b-492e-4092-9889-c2835f58b761-plugins-config-volume" pod "514c415b-492e-4092-9889-c2835f58b761" (UID: "514c415b-492e-4092-9889-c2835f58b761"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:55:21.693119 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:21.693077    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/514c415b-492e-4092-9889-c2835f58b761-kube-api-access-dzm8s" pod "514c415b-492e-4092-9889-c2835f58b761" (UID: "514c415b-492e-4092-9889-c2835f58b761"). InnerVolumeSpecName "kube-api-access-dzm8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:55:21.768212 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:21.768179    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-86d620-fb8d7795f-mjmqg"
Oct 30 20:55:21.784943 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:21.784911    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/904e938e-ad13-4460-9ab6-b1ce6ed3c1d6-kube-api-access-jdcth\" (UniqueName: \"kubernetes.io/projected/904e938e-ad13-4460-9ab6-b1ce6ed3c1d6-kube-api-access-jdcth\") pod \"904e938e-ad13-4460-9ab6-b1ce6ed3c1d6\" (UID: \"904e938e-ad13-4460-9ab6-b1ce6ed3c1d6\") "
Oct 30 20:55:21.785313 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:21.785290    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/904e938e-ad13-4460-9ab6-b1ce6ed3c1d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/904e938e-ad13-4460-9ab6-b1ce6ed3c1d6-plugins-config-volume\") pod \"904e938e-ad13-4460-9ab6-b1ce6ed3c1d6\" (UID: \"904e938e-ad13-4460-9ab6-b1ce6ed3c1d6\") "
Oct 30 20:55:21.786036 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:21.786013    5071 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/514c415b-492e-4092-9889-c2835f58b761-plugins-config-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:21.786177 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:21.786160    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzm8s\" (UniqueName: \"kubernetes.io/projected/514c415b-492e-4092-9889-c2835f58b761-kube-api-access-dzm8s\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:21.786267 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:21.786020    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/904e938e-ad13-4460-9ab6-b1ce6ed3c1d6-plugins-config-volume" pod "904e938e-ad13-4460-9ab6-b1ce6ed3c1d6" (UID: "904e938e-ad13-4460-9ab6-b1ce6ed3c1d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:55:21.787214 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:21.787190    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/904e938e-ad13-4460-9ab6-b1ce6ed3c1d6-kube-api-access-jdcth" pod "904e938e-ad13-4460-9ab6-b1ce6ed3c1d6" (UID: "904e938e-ad13-4460-9ab6-b1ce6ed3c1d6"). InnerVolumeSpecName "kube-api-access-jdcth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:55:21.887512 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:21.887476    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdcth\" (UniqueName: \"kubernetes.io/projected/904e938e-ad13-4460-9ab6-b1ce6ed3c1d6-kube-api-access-jdcth\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:21.887512 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:21.887515    5071 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/904e938e-ad13-4460-9ab6-b1ce6ed3c1d6-plugins-config-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:22.184906 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:22.184864    5071 generic.go:356] "Generic (PLEG): container finished" podID="904e938e-ad13-4460-9ab6-b1ce6ed3c1d6" containerID="4c9dfe7e186d197b98d166719328a4df91bcb0319732c03a60124dcc9c986061" exitCode=0
Oct 30 20:55:22.185397 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:22.184923    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-86d620-fb8d7795f-mjmqg"
Oct 30 20:55:22.185397 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:22.184995    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-86d620-fb8d7795f-mjmqg" event={"ID":"904e938e-ad13-4460-9ab6-b1ce6ed3c1d6","Type":"ContainerDied","Data":"4c9dfe7e186d197b98d166719328a4df91bcb0319732c03a60124dcc9c986061"}
Oct 30 20:55:22.185397 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:22.185089    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-86d620-fb8d7795f-mjmqg" event={"ID":"904e938e-ad13-4460-9ab6-b1ce6ed3c1d6","Type":"ContainerDied","Data":"142f4444ab38adf16c1d2578b6934b10a850d236f322845a2fc85277d904b099"}
Oct 30 20:55:22.185397 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:22.185125    5071 scope.go:122] "RemoveContainer" containerID="4c9dfe7e186d197b98d166719328a4df91bcb0319732c03a60124dcc9c986061"
Oct 30 20:55:22.189460 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:22.189436    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd" event={"ID":"b1203632-126c-4620-8740-6f07120ad6cd","Type":"ContainerStarted","Data":"d1c6b4e6aa9d21e6cd5c2eb4e1e0479016f7e94d2c1a5453f58424bf038c34fa"}
Oct 30 20:55:22.189563 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:22.189509    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd"
Oct 30 20:55:22.191093 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:22.191069    5071 generic.go:356] "Generic (PLEG): container finished" podID="514c415b-492e-4092-9889-c2835f58b761" containerID="1df1e4d8ec1594c9a65542c5944f756af17a466de11a134503ed67b3f99765a4" exitCode=0
Oct 30 20:55:22.191207 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:22.191164    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-86d620-74c5fd4b96-2zkm5"
Oct 30 20:55:22.191275 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:22.191207    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-86d620-74c5fd4b96-2zkm5" event={"ID":"514c415b-492e-4092-9889-c2835f58b761","Type":"ContainerDied","Data":"1df1e4d8ec1594c9a65542c5944f756af17a466de11a134503ed67b3f99765a4"}
Oct 30 20:55:22.191275 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:22.191254    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-86d620-74c5fd4b96-2zkm5" event={"ID":"514c415b-492e-4092-9889-c2835f58b761","Type":"ContainerDied","Data":"ae95b8ac112aee23df6a129ea087ff9439b7a7d200c35bd0e201b0d8b7a590ce"}
Oct 30 20:55:22.202221 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:22.202026    5071 scope.go:122] "RemoveContainer" containerID="4c9dfe7e186d197b98d166719328a4df91bcb0319732c03a60124dcc9c986061"
Oct 30 20:55:22.202353 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:55:22.202332    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c9dfe7e186d197b98d166719328a4df91bcb0319732c03a60124dcc9c986061\": container with ID starting with 4c9dfe7e186d197b98d166719328a4df91bcb0319732c03a60124dcc9c986061 not found: ID does not exist" containerID="4c9dfe7e186d197b98d166719328a4df91bcb0319732c03a60124dcc9c986061"
Oct 30 20:55:22.202395 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:22.202361    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c9dfe7e186d197b98d166719328a4df91bcb0319732c03a60124dcc9c986061"} err="failed to get container status \"4c9dfe7e186d197b98d166719328a4df91bcb0319732c03a60124dcc9c986061\": rpc error: code = NotFound desc = could not find container \"4c9dfe7e186d197b98d166719328a4df91bcb0319732c03a60124dcc9c986061\": container with ID starting with 4c9dfe7e186d197b98d166719328a4df91bcb0319732c03a60124dcc9c986061 not found: ID does not exist"
Oct 30 20:55:22.202395 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:22.202380    5071 scope.go:122] "RemoveContainer" containerID="1df1e4d8ec1594c9a65542c5944f756af17a466de11a134503ed67b3f99765a4"
Oct 30 20:55:22.300863 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:22.298595    5071 scope.go:122] "RemoveContainer" containerID="1df1e4d8ec1594c9a65542c5944f756af17a466de11a134503ed67b3f99765a4"
Oct 30 20:55:22.306021 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:55:22.305986    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1df1e4d8ec1594c9a65542c5944f756af17a466de11a134503ed67b3f99765a4\": container with ID starting with 1df1e4d8ec1594c9a65542c5944f756af17a466de11a134503ed67b3f99765a4 not found: ID does not exist" containerID="1df1e4d8ec1594c9a65542c5944f756af17a466de11a134503ed67b3f99765a4"
Oct 30 20:55:22.306378 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:22.306331    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1df1e4d8ec1594c9a65542c5944f756af17a466de11a134503ed67b3f99765a4"} err="failed to get container status \"1df1e4d8ec1594c9a65542c5944f756af17a466de11a134503ed67b3f99765a4\": rpc error: code = NotFound desc = could not find container \"1df1e4d8ec1594c9a65542c5944f756af17a466de11a134503ed67b3f99765a4\": container with ID starting with 1df1e4d8ec1594c9a65542c5944f756af17a466de11a134503ed67b3f99765a4 not found: ID does not exist"
Oct 30 20:55:22.369838 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:22.369754    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd" podStartSLOduration=3.5063983199999997 podStartE2EDuration="4.369734189s" totalImagesPullingTime="863.335869ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:22.368396858 +0000 UTC m=+1734.285418395" watchObservedRunningTime="2026-10-30 20:55:22.369734189 +0000 UTC m=+1734.286755726"
Oct 30 20:55:22.495901 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:22.492256    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-86d620-74c5fd4b96-2zkm5"]
Oct 30 20:55:22.503347 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:22.502383    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-86d620-74c5fd4b96-2zkm5"]
Oct 30 20:55:22.520264 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:22.520228    5071 patch_prober.go:29] interesting pod/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 20:55:22.520556 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:22.520523    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt" podUID="d204447d-7147-415b-abf8-853f68b51fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 20:55:22.612784 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:22.612747    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-86d620-fb8d7795f-mjmqg"]
Oct 30 20:55:22.617737 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:22.617717    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-86d620-fb8d7795f-mjmqg"]
Oct 30 20:55:23.518737 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:23.518696    5071 patch_prober.go:29] interesting pod/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 20:55:23.519158 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:23.518757    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt" podUID="d204447d-7147-415b-abf8-853f68b51fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 20:55:24.002585 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:24.002551    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:24.208659 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:24.208624    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="514c415b-492e-4092-9889-c2835f58b761" path="/var/lib/kubelet/pods/514c415b-492e-4092-9889-c2835f58b761/volumes"
Oct 30 20:55:24.209113 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:24.209098    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="904e938e-ad13-4460-9ab6-b1ce6ed3c1d6" path="/var/lib/kubelet/pods/904e938e-ad13-4460-9ab6-b1ce6ed3c1d6/volumes"
Oct 30 20:55:24.518575 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:24.518544    5071 patch_prober.go:29] interesting pod/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 20:55:24.518770 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:24.518595    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt" podUID="d204447d-7147-415b-abf8-853f68b51fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 20:55:25.519673 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:25.519201    5071 patch_prober.go:29] interesting pod/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 20:55:25.519673 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:25.519277    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt" podUID="d204447d-7147-415b-abf8-853f68b51fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 20:55:25.860993 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:25.860957    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:26.152905 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:26.148422    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:26.322698 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:26.322663    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:26.618143 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:26.618114    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:55:26.619116 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:26.619092    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:55:26.619969 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:26.619925    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:55:26.794205 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:26.794163    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:26.968478 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:26.968272    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:28.120557 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:28.120515    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:28.208711 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:28.208682    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd"
Oct 30 20:55:30.056256 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:30.056208    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-939d410e-twdgv"]
Oct 30 20:55:30.105688 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:30.105658    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-939d410e-twdgv"]
Oct 30 20:55:30.203725 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:30.203692    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd" path="/var/lib/kubelet/pods/c71df5c9-8eae-47f4-bbbb-2c1ddb1697dd/volumes"
Oct 30 20:55:31.084777 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:31.084737    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:55:31.344892 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:31.344451    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w"]
Oct 30 20:55:31.346228 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:31.346186    5071 cpu_manager.go:414] "containerMap: removing container" podUID="904e938e-ad13-4460-9ab6-b1ce6ed3c1d6" containerName="payload-pre-processing"
Oct 30 20:55:31.351897 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:31.351868    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="904e938e-ad13-4460-9ab6-b1ce6ed3c1d6" containerName="payload-pre-processing"
Oct 30 20:55:31.352107 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:31.352088    5071 cpu_manager.go:414] "containerMap: removing container" podUID="514c415b-492e-4092-9889-c2835f58b761" containerName="payload-processing"
Oct 30 20:55:31.352196 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:31.352184    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="514c415b-492e-4092-9889-c2835f58b761" containerName="payload-processing"
Oct 30 20:55:31.352879 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:31.352844    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="904e938e-ad13-4460-9ab6-b1ce6ed3c1d6" containerName="payload-pre-processing"
Oct 30 20:55:31.355784 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:31.355764    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="514c415b-492e-4092-9889-c2835f58b761" containerName="payload-processing"
Oct 30 20:55:31.405791 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:31.405752    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w"
Oct 30 20:55:31.406451 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:31.406428    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w"]
Oct 30 20:55:31.459182 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:31.459152    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-d9b3f6-serving-cert\""
Oct 30 20:55:31.512689 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:31.512653    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:31.565945 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:31.565898    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c628k\" (UniqueName: \"kubernetes.io/projected/e830ac30-2863-47d3-a501-5d0622dccb81-kube-api-access-c628k\") pod \"maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w\" (UID: \"e830ac30-2863-47d3-a501-5d0622dccb81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w"
Oct 30 20:55:31.577655 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:31.566684    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e830ac30-2863-47d3-a501-5d0622dccb81-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w\" (UID: \"e830ac30-2863-47d3-a501-5d0622dccb81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w"
Oct 30 20:55:31.577655 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:31.569170    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e830ac30-2863-47d3-a501-5d0622dccb81-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w\" (UID: \"e830ac30-2863-47d3-a501-5d0622dccb81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w"
Oct 30 20:55:31.577655 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:31.569280    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e830ac30-2863-47d3-a501-5d0622dccb81-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w\" (UID: \"e830ac30-2863-47d3-a501-5d0622dccb81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w"
Oct 30 20:55:31.671069 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:31.670985    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e830ac30-2863-47d3-a501-5d0622dccb81-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w\" (UID: \"e830ac30-2863-47d3-a501-5d0622dccb81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w"
Oct 30 20:55:31.671069 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:31.671045    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e830ac30-2863-47d3-a501-5d0622dccb81-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w\" (UID: \"e830ac30-2863-47d3-a501-5d0622dccb81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w"
Oct 30 20:55:31.671255 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:31.671190    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c628k\" (UniqueName: \"kubernetes.io/projected/e830ac30-2863-47d3-a501-5d0622dccb81-kube-api-access-c628k\") pod \"maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w\" (UID: \"e830ac30-2863-47d3-a501-5d0622dccb81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w"
Oct 30 20:55:31.671255 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:31.671236    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e830ac30-2863-47d3-a501-5d0622dccb81-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w\" (UID: \"e830ac30-2863-47d3-a501-5d0622dccb81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w"
Oct 30 20:55:31.671814 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:31.671791    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e830ac30-2863-47d3-a501-5d0622dccb81-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w\" (UID: \"e830ac30-2863-47d3-a501-5d0622dccb81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w"
Oct 30 20:55:31.673759 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:31.673728    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e830ac30-2863-47d3-a501-5d0622dccb81-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w\" (UID: \"e830ac30-2863-47d3-a501-5d0622dccb81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w"
Oct 30 20:55:31.673951 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:31.673933    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e830ac30-2863-47d3-a501-5d0622dccb81-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w\" (UID: \"e830ac30-2863-47d3-a501-5d0622dccb81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w"
Oct 30 20:55:31.735724 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:31.735687    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:31.737044 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:31.737016    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c628k\" (UniqueName: \"kubernetes.io/projected/e830ac30-2863-47d3-a501-5d0622dccb81-kube-api-access-c628k\") pod \"maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w\" (UID: \"e830ac30-2863-47d3-a501-5d0622dccb81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w"
Oct 30 20:55:32.027177 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:32.027102    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w"
Oct 30 20:55:32.492775 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:32.492743    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:32.597069 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:32.597033    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w"]
Oct 30 20:55:32.597186 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:55:32.597033    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode830ac30_2863_47d3_a501_5d0622dccb81.slice/crio-c56249508f3922492ccae759f5229982a635748b482cae119588c92e7661441c WatchSource:0}: Error finding container c56249508f3922492ccae759f5229982a635748b482cae119588c92e7661441c: Status 404 returned error can't find the container with id c56249508f3922492ccae759f5229982a635748b482cae119588c92e7661441c
Oct 30 20:55:33.429592 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:33.429555    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w" event={"ID":"e830ac30-2863-47d3-a501-5d0622dccb81","Type":"ContainerStarted","Data":"c56249508f3922492ccae759f5229982a635748b482cae119588c92e7661441c"}
Oct 30 20:55:33.981423 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:33.981387    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:34.197497 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:34.197460    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:34.439226 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:34.439187    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w" event={"ID":"e830ac30-2863-47d3-a501-5d0622dccb81","Type":"ContainerStarted","Data":"e94670734adcd150dffab1ec688fdc4e7e1b4cd1b1a46ffc37aaab998c7530a0"}
Oct 30 20:55:34.439422 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:34.439257    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w"
Oct 30 20:55:34.556241 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:34.556181    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w" podStartSLOduration=2.65419663 podStartE2EDuration="3.556164828s" totalImagesPullingTime="901.968198ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:34.554987947 +0000 UTC m=+1746.472009497" watchObservedRunningTime="2026-10-30 20:55:34.556164828 +0000 UTC m=+1746.473186366"
Oct 30 20:55:34.882573 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:34.882537    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:35.188538 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:35.188452    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:35.506546 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:35.506458    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:36.198208 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:36.198168    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:36.370318 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:36.370282    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:36.495490 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:36.495413    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"]
Oct 30 20:55:36.495732 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:36.495702    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6" podUID="99069007-e0f8-4092-8fd6-f27b0a119720" containerName="istio-proxy" containerID="cri-o://6cdaafbc6f4f2066fb222068e8e940e1b0214f283427088b744a0bac7802766e" gracePeriod=30
Oct 30 20:55:36.547529 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:36.547489    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:37.038584 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:37.038486    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:37.226661 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:37.226565    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:37.400094 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:37.400011    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:38.385047 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:38.385013    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:40.460394 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:40.460367    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w"
Oct 30 20:55:41.789027 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:41.789007    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:55:41.884593 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:41.884564    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:41.897240 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:41.897215    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99069007-e0f8-4092-8fd6-f27b0a119720-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99069007-e0f8-4092-8fd6-f27b0a119720-credential-socket\") pod \"99069007-e0f8-4092-8fd6-f27b0a119720\" (UID: \"99069007-e0f8-4092-8fd6-f27b0a119720\") "
Oct 30 20:55:41.897381 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:41.897327    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99069007-e0f8-4092-8fd6-f27b0a119720-kube-api-access-h7frs\" (UniqueName: \"kubernetes.io/projected/99069007-e0f8-4092-8fd6-f27b0a119720-kube-api-access-h7frs\") pod \"99069007-e0f8-4092-8fd6-f27b0a119720\" (UID: \"99069007-e0f8-4092-8fd6-f27b0a119720\") "
Oct 30 20:55:41.897451 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:41.897428    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99069007-e0f8-4092-8fd6-f27b0a119720-istio-token\" (UniqueName: \"kubernetes.io/projected/99069007-e0f8-4092-8fd6-f27b0a119720-istio-token\") pod \"99069007-e0f8-4092-8fd6-f27b0a119720\" (UID: \"99069007-e0f8-4092-8fd6-f27b0a119720\") "
Oct 30 20:55:41.897505 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:41.897474    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99069007-e0f8-4092-8fd6-f27b0a119720-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99069007-e0f8-4092-8fd6-f27b0a119720-istio-envoy\") pod \"99069007-e0f8-4092-8fd6-f27b0a119720\" (UID: \"99069007-e0f8-4092-8fd6-f27b0a119720\") "
Oct 30 20:55:41.897505 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:41.897474    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99069007-e0f8-4092-8fd6-f27b0a119720-credential-socket" pod "99069007-e0f8-4092-8fd6-f27b0a119720" (UID: "99069007-e0f8-4092-8fd6-f27b0a119720"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:55:41.897631 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:41.897527    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99069007-e0f8-4092-8fd6-f27b0a119720-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99069007-e0f8-4092-8fd6-f27b0a119720-workload-socket\") pod \"99069007-e0f8-4092-8fd6-f27b0a119720\" (UID: \"99069007-e0f8-4092-8fd6-f27b0a119720\") "
Oct 30 20:55:41.897631 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:41.897603    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99069007-e0f8-4092-8fd6-f27b0a119720-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99069007-e0f8-4092-8fd6-f27b0a119720-istio-podinfo\") pod \"99069007-e0f8-4092-8fd6-f27b0a119720\" (UID: \"99069007-e0f8-4092-8fd6-f27b0a119720\") "
Oct 30 20:55:41.897751 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:41.897644    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99069007-e0f8-4092-8fd6-f27b0a119720-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99069007-e0f8-4092-8fd6-f27b0a119720-istio-data\") pod \"99069007-e0f8-4092-8fd6-f27b0a119720\" (UID: \"99069007-e0f8-4092-8fd6-f27b0a119720\") "
Oct 30 20:55:41.897751 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:41.897673    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99069007-e0f8-4092-8fd6-f27b0a119720-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99069007-e0f8-4092-8fd6-f27b0a119720-istiod-ca-cert\") pod \"99069007-e0f8-4092-8fd6-f27b0a119720\" (UID: \"99069007-e0f8-4092-8fd6-f27b0a119720\") "
Oct 30 20:55:41.897751 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:41.897728    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99069007-e0f8-4092-8fd6-f27b0a119720-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99069007-e0f8-4092-8fd6-f27b0a119720-workload-certs\") pod \"99069007-e0f8-4092-8fd6-f27b0a119720\" (UID: \"99069007-e0f8-4092-8fd6-f27b0a119720\") "
Oct 30 20:55:41.897966 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:41.897900    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99069007-e0f8-4092-8fd6-f27b0a119720-workload-socket" pod "99069007-e0f8-4092-8fd6-f27b0a119720" (UID: "99069007-e0f8-4092-8fd6-f27b0a119720"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:55:41.897966 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:41.897937    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99069007-e0f8-4092-8fd6-f27b0a119720-istio-data" pod "99069007-e0f8-4092-8fd6-f27b0a119720" (UID: "99069007-e0f8-4092-8fd6-f27b0a119720"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:55:41.898283 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:41.898150    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99069007-e0f8-4092-8fd6-f27b0a119720-workload-certs" pod "99069007-e0f8-4092-8fd6-f27b0a119720" (UID: "99069007-e0f8-4092-8fd6-f27b0a119720"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:55:41.898283 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:41.898184    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99069007-e0f8-4092-8fd6-f27b0a119720-istiod-ca-cert" pod "99069007-e0f8-4092-8fd6-f27b0a119720" (UID: "99069007-e0f8-4092-8fd6-f27b0a119720"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:55:41.898654 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:41.898624    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99069007-e0f8-4092-8fd6-f27b0a119720-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:41.898735 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:41.898661    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99069007-e0f8-4092-8fd6-f27b0a119720-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:41.898735 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:41.898677    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99069007-e0f8-4092-8fd6-f27b0a119720-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:41.898735 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:41.898692    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99069007-e0f8-4092-8fd6-f27b0a119720-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:41.898735 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:41.898706    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99069007-e0f8-4092-8fd6-f27b0a119720-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:41.899918 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:41.899891    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99069007-e0f8-4092-8fd6-f27b0a119720-kube-api-access-h7frs" pod "99069007-e0f8-4092-8fd6-f27b0a119720" (UID: "99069007-e0f8-4092-8fd6-f27b0a119720"). InnerVolumeSpecName "kube-api-access-h7frs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:55:41.899918 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:41.899908    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99069007-e0f8-4092-8fd6-f27b0a119720-istio-token" pod "99069007-e0f8-4092-8fd6-f27b0a119720" (UID: "99069007-e0f8-4092-8fd6-f27b0a119720"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:55:41.900077 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:41.899950    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99069007-e0f8-4092-8fd6-f27b0a119720-istio-envoy" pod "99069007-e0f8-4092-8fd6-f27b0a119720" (UID: "99069007-e0f8-4092-8fd6-f27b0a119720"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:55:41.900077 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:41.900040    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99069007-e0f8-4092-8fd6-f27b0a119720-istio-podinfo" pod "99069007-e0f8-4092-8fd6-f27b0a119720" (UID: "99069007-e0f8-4092-8fd6-f27b0a119720"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:55:42.000224 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:42.000135    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99069007-e0f8-4092-8fd6-f27b0a119720-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:42.000224 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:42.000168    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7frs\" (UniqueName: \"kubernetes.io/projected/99069007-e0f8-4092-8fd6-f27b0a119720-kube-api-access-h7frs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:42.000224 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:42.000186    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99069007-e0f8-4092-8fd6-f27b0a119720-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:42.000224 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:42.000200    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99069007-e0f8-4092-8fd6-f27b0a119720-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:42.159578 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:42.159501    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:42.519486 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:42.519391    5071 generic.go:356] "Generic (PLEG): container finished" podID="99069007-e0f8-4092-8fd6-f27b0a119720" containerID="6cdaafbc6f4f2066fb222068e8e940e1b0214f283427088b744a0bac7802766e" exitCode=0
Oct 30 20:55:42.519486 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:42.519433    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"
Oct 30 20:55:42.519486 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:42.519446    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6" event={"ID":"99069007-e0f8-4092-8fd6-f27b0a119720","Type":"ContainerDied","Data":"6cdaafbc6f4f2066fb222068e8e940e1b0214f283427088b744a0bac7802766e"}
Oct 30 20:55:42.519762 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:42.519495    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6" event={"ID":"99069007-e0f8-4092-8fd6-f27b0a119720","Type":"ContainerDied","Data":"3d9a041f137d200cfbbaddcd5be13196b399318015f7020ab1a331c3664520be"}
Oct 30 20:55:42.519762 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:42.519518    5071 scope.go:122] "RemoveContainer" containerID="6cdaafbc6f4f2066fb222068e8e940e1b0214f283427088b744a0bac7802766e"
Oct 30 20:55:42.532191 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:42.532169    5071 scope.go:122] "RemoveContainer" containerID="63d8251f784c6fab20cf2a5ba32c42a286a091cc3a95f12eff1c80c5ddef01db"
Oct 30 20:55:42.543225 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:42.543195    5071 scope.go:122] "RemoveContainer" containerID="6cdaafbc6f4f2066fb222068e8e940e1b0214f283427088b744a0bac7802766e"
Oct 30 20:55:42.543525 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:55:42.543493    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cdaafbc6f4f2066fb222068e8e940e1b0214f283427088b744a0bac7802766e\": container with ID starting with 6cdaafbc6f4f2066fb222068e8e940e1b0214f283427088b744a0bac7802766e not found: ID does not exist" containerID="6cdaafbc6f4f2066fb222068e8e940e1b0214f283427088b744a0bac7802766e"
Oct 30 20:55:42.543632 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:42.543534    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cdaafbc6f4f2066fb222068e8e940e1b0214f283427088b744a0bac7802766e"} err="failed to get container status \"6cdaafbc6f4f2066fb222068e8e940e1b0214f283427088b744a0bac7802766e\": rpc error: code = NotFound desc = could not find container \"6cdaafbc6f4f2066fb222068e8e940e1b0214f283427088b744a0bac7802766e\": container with ID starting with 6cdaafbc6f4f2066fb222068e8e940e1b0214f283427088b744a0bac7802766e not found: ID does not exist"
Oct 30 20:55:42.543632 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:42.543564    5071 scope.go:122] "RemoveContainer" containerID="63d8251f784c6fab20cf2a5ba32c42a286a091cc3a95f12eff1c80c5ddef01db"
Oct 30 20:55:42.543868 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:55:42.543848    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63d8251f784c6fab20cf2a5ba32c42a286a091cc3a95f12eff1c80c5ddef01db\": container with ID starting with 63d8251f784c6fab20cf2a5ba32c42a286a091cc3a95f12eff1c80c5ddef01db not found: ID does not exist" containerID="63d8251f784c6fab20cf2a5ba32c42a286a091cc3a95f12eff1c80c5ddef01db"
Oct 30 20:55:42.543952 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:42.543876    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63d8251f784c6fab20cf2a5ba32c42a286a091cc3a95f12eff1c80c5ddef01db"} err="failed to get container status \"63d8251f784c6fab20cf2a5ba32c42a286a091cc3a95f12eff1c80c5ddef01db\": rpc error: code = NotFound desc = could not find container \"63d8251f784c6fab20cf2a5ba32c42a286a091cc3a95f12eff1c80c5ddef01db\": container with ID starting with 63d8251f784c6fab20cf2a5ba32c42a286a091cc3a95f12eff1c80c5ddef01db not found: ID does not exist"
Oct 30 20:55:42.738912 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:42.738877    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"]
Oct 30 20:55:42.788518 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:42.788490    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b32941-openshift-default-649798b65f-v4fh6"]
Oct 30 20:55:44.205068 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:44.203407    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99069007-e0f8-4092-8fd6-f27b0a119720" path="/var/lib/kubelet/pods/99069007-e0f8-4092-8fd6-f27b0a119720/volumes"
Oct 30 20:55:46.731534 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:55:46.731492    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:55:46.767632 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:46.767602    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:51.623092 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:51.623046    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:55:56.997117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:55:56.997075    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:01.102683 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:01.102638    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:02.273073 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:02.273034    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:03.283139 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:03.283107    5071 patch_prober.go:29] interesting pod/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 20:56:03.283511 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:03.283170    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4" podUID="79131253-b9a6-41dc-b1d4-aa525c4c5e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 20:56:03.769773 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:03.769730    5071 generic.go:356] "Generic (PLEG): container finished" podID="2c36da32-7ded-4d53-8579-f62445fc61a5" containerID="7c11413bb0d8e2fdbfdd114869b7450d0a54292162df6de3363659ca73b68163" exitCode=0
Oct 30 20:56:03.770001 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:03.769839    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw" event={"ID":"2c36da32-7ded-4d53-8579-f62445fc61a5","Type":"ContainerDied","Data":"7c11413bb0d8e2fdbfdd114869b7450d0a54292162df6de3363659ca73b68163"}
Oct 30 20:56:03.770327 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:03.770304    5071 scope.go:122] "RemoveContainer" containerID="7c11413bb0d8e2fdbfdd114869b7450d0a54292162df6de3363659ca73b68163"
Oct 30 20:56:03.772120 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:03.772088    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:56:03.772254 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:03.772169    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:56:03.772254 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:03.772216    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:56:03.772539 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:03.772517    5071 generic.go:356] "Generic (PLEG): container finished" podID="79131253-b9a6-41dc-b1d4-aa525c4c5e04" containerID="5a6213b9ad5df080a05d78effb7cc46f8bc54cec21558f29168e3cec6a358d83" exitCode=0
Oct 30 20:56:03.772606 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:03.772588    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4" event={"ID":"79131253-b9a6-41dc-b1d4-aa525c4c5e04","Type":"ContainerDied","Data":"5a6213b9ad5df080a05d78effb7cc46f8bc54cec21558f29168e3cec6a358d83"}
Oct 30 20:56:03.772996 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:03.772980    5071 scope.go:122] "RemoveContainer" containerID="5a6213b9ad5df080a05d78effb7cc46f8bc54cec21558f29168e3cec6a358d83"
Oct 30 20:56:03.774759 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:03.774719    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:56:03.774895 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:03.774796    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:56:03.774895 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:03.774850    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:56:04.795025 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:04.794954    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw" event={"ID":"2c36da32-7ded-4d53-8579-f62445fc61a5","Type":"ContainerStarted","Data":"15cdf37757674bf07863987b38358fb28ad8564879f10999a9c3ca279dcdf069"}
Oct 30 20:56:04.797320 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:04.797290    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4" event={"ID":"79131253-b9a6-41dc-b1d4-aa525c4c5e04","Type":"ContainerStarted","Data":"4c71836aa50f8860426f4b7d3f4c0d274b4e1cc86238239677e630b45fa00f24"}
Oct 30 20:56:05.196879 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:05.196789    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:56:05.197031 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:05.196899    5071 patch_prober.go:29] interesting pod/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 20:56:05.197031 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:05.196933    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4" podUID="79131253-b9a6-41dc-b1d4-aa525c4c5e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 20:56:05.453265 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:05.451201    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 20:56:05.453265 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:05.452864    5071 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 30 20:56:05.453265 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:05.452920    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw" podUID="2c36da32-7ded-4d53-8579-f62445fc61a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 30 20:56:06.196107 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:06.196071    5071 patch_prober.go:29] interesting pod/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 20:56:06.196491 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:06.196129    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4" podUID="79131253-b9a6-41dc-b1d4-aa525c4c5e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 20:56:06.461902 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:06.461159    5071 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 30 20:56:06.461902 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:06.461221    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw" podUID="2c36da32-7ded-4d53-8579-f62445fc61a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 30 20:56:07.196170 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:07.196135    5071 patch_prober.go:29] interesting pod/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 20:56:07.196701 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:07.196246    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4" podUID="79131253-b9a6-41dc-b1d4-aa525c4c5e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 20:56:07.454433 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:07.454333    5071 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 30 20:56:07.454855 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:07.454804    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw" podUID="2c36da32-7ded-4d53-8579-f62445fc61a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 30 20:56:07.695939 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:07.695897    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:08.201949 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:08.200744    5071 patch_prober.go:29] interesting pod/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 20:56:08.201949 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:08.200789    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4" podUID="79131253-b9a6-41dc-b1d4-aa525c4c5e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 20:56:08.452244 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:08.452146    5071 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 30 20:56:08.452519 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:08.452489    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw" podUID="2c36da32-7ded-4d53-8579-f62445fc61a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 30 20:56:08.658848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:08.654907    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 20:56:09.198543 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:09.198068    5071 patch_prober.go:29] interesting pod/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 20:56:09.198543 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:09.198127    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4" podUID="79131253-b9a6-41dc-b1d4-aa525c4c5e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 20:56:09.453899 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:09.453165    5071 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 30 20:56:09.453899 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:09.453225    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw" podUID="2c36da32-7ded-4d53-8579-f62445fc61a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 30 20:56:10.208550 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:10.208505    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:56:10.209227 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:10.209203    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:56:10.210601 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:10.210580    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:56:10.455303 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:10.455264    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 20:56:10.456517 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:10.456495    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 20:56:13.287709 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:13.287667    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:14.261220 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:14.261188    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:56:14.881695 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:14.881664    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b86743-7c48bdc49b-xdsk6"]
Oct 30 20:56:14.997039 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:14.997006    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b86743-7fbb475c6c-gfq7v"]
Oct 30 20:56:15.372492 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:15.372398    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:15.540001 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:15.539969    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:16.867601 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:56:16.867568    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:56:18.622899 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:18.622862    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:18.931900 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:18.931778    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:19.109240 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:19.109207    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:19.882941 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:19.882889    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b86743-7c48bdc49b-xdsk6" podUID="4d3048f3-fbc4-4718-ae16-55b2c6c54890" containerName="payload-processing" containerID="cri-o://b1660487fa13d855e4da0aa4d9be71b57ac9154452d88d9a939bb15b5a0712c1" gracePeriod=25
Oct 30 20:56:20.001978 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.001936    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b86743-7fbb475c6c-gfq7v" podUID="9b4b151c-97ec-4a24-9569-53eba1c22d79" containerName="payload-pre-processing" containerID="cri-o://fd9fddd9a7fba0ded36c2012a5bf760c4a96a03ffa040aa4c05ad5d2d4e5575e" gracePeriod=25
Oct 30 20:56:20.033619 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.033587    5071 generic.go:356] "Generic (PLEG): container finished" podID="4d3048f3-fbc4-4718-ae16-55b2c6c54890" containerID="b1660487fa13d855e4da0aa4d9be71b57ac9154452d88d9a939bb15b5a0712c1" exitCode=0
Oct 30 20:56:20.033767 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.033654    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b86743-7c48bdc49b-xdsk6" event={"ID":"4d3048f3-fbc4-4718-ae16-55b2c6c54890","Type":"ContainerDied","Data":"b1660487fa13d855e4da0aa4d9be71b57ac9154452d88d9a939bb15b5a0712c1"}
Oct 30 20:56:20.200524 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.200384    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b86743-7c48bdc49b-xdsk6"
Oct 30 20:56:20.317165 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.317137    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d3048f3-fbc4-4718-ae16-55b2c6c54890-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d3048f3-fbc4-4718-ae16-55b2c6c54890-plugins-config-volume\") pod \"4d3048f3-fbc4-4718-ae16-55b2c6c54890\" (UID: \"4d3048f3-fbc4-4718-ae16-55b2c6c54890\") "
Oct 30 20:56:20.317317 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.317183    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d3048f3-fbc4-4718-ae16-55b2c6c54890-kube-api-access-xmhxg\" (UniqueName: \"kubernetes.io/projected/4d3048f3-fbc4-4718-ae16-55b2c6c54890-kube-api-access-xmhxg\") pod \"4d3048f3-fbc4-4718-ae16-55b2c6c54890\" (UID: \"4d3048f3-fbc4-4718-ae16-55b2c6c54890\") "
Oct 30 20:56:20.317567 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.317540    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d3048f3-fbc4-4718-ae16-55b2c6c54890-plugins-config-volume" pod "4d3048f3-fbc4-4718-ae16-55b2c6c54890" (UID: "4d3048f3-fbc4-4718-ae16-55b2c6c54890"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:56:20.317787 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.317770    5071 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d3048f3-fbc4-4718-ae16-55b2c6c54890-plugins-config-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:20.319924 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.319888    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d3048f3-fbc4-4718-ae16-55b2c6c54890-kube-api-access-xmhxg" pod "4d3048f3-fbc4-4718-ae16-55b2c6c54890" (UID: "4d3048f3-fbc4-4718-ae16-55b2c6c54890"). InnerVolumeSpecName "kube-api-access-xmhxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:56:20.321053 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.321027    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b86743-7fbb475c6c-gfq7v"
Oct 30 20:56:20.418336 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.418293    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b4b151c-97ec-4a24-9569-53eba1c22d79-kube-api-access-jkz4s\" (UniqueName: \"kubernetes.io/projected/9b4b151c-97ec-4a24-9569-53eba1c22d79-kube-api-access-jkz4s\") pod \"9b4b151c-97ec-4a24-9569-53eba1c22d79\" (UID: \"9b4b151c-97ec-4a24-9569-53eba1c22d79\") "
Oct 30 20:56:20.418559 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.418379    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b4b151c-97ec-4a24-9569-53eba1c22d79-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b4b151c-97ec-4a24-9569-53eba1c22d79-plugins-config-volume\") pod \"9b4b151c-97ec-4a24-9569-53eba1c22d79\" (UID: \"9b4b151c-97ec-4a24-9569-53eba1c22d79\") "
Oct 30 20:56:20.419093 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.419066    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmhxg\" (UniqueName: \"kubernetes.io/projected/4d3048f3-fbc4-4718-ae16-55b2c6c54890-kube-api-access-xmhxg\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:20.419093 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.419066    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b4b151c-97ec-4a24-9569-53eba1c22d79-plugins-config-volume" pod "9b4b151c-97ec-4a24-9569-53eba1c22d79" (UID: "9b4b151c-97ec-4a24-9569-53eba1c22d79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:56:20.421233 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.421192    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b4b151c-97ec-4a24-9569-53eba1c22d79-kube-api-access-jkz4s" pod "9b4b151c-97ec-4a24-9569-53eba1c22d79" (UID: "9b4b151c-97ec-4a24-9569-53eba1c22d79"). InnerVolumeSpecName "kube-api-access-jkz4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:56:20.431923 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.431894    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"]
Oct 30 20:56:20.433770 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.433747    5071 cpu_manager.go:414] "containerMap: removing container" podUID="9b4b151c-97ec-4a24-9569-53eba1c22d79" containerName="payload-pre-processing"
Oct 30 20:56:20.433926 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.433773    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b4b151c-97ec-4a24-9569-53eba1c22d79" containerName="payload-pre-processing"
Oct 30 20:56:20.433926 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.433795    5071 cpu_manager.go:414] "containerMap: removing container" podUID="4d3048f3-fbc4-4718-ae16-55b2c6c54890" containerName="payload-processing"
Oct 30 20:56:20.433926 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.433803    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d3048f3-fbc4-4718-ae16-55b2c6c54890" containerName="payload-processing"
Oct 30 20:56:20.433926 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.433854    5071 cpu_manager.go:414] "containerMap: removing container" podUID="99069007-e0f8-4092-8fd6-f27b0a119720" containerName="istio-proxy"
Oct 30 20:56:20.433926 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.433862    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99069007-e0f8-4092-8fd6-f27b0a119720" containerName="istio-proxy"
Oct 30 20:56:20.433926 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.433886    5071 cpu_manager.go:414] "containerMap: removing container" podUID="99069007-e0f8-4092-8fd6-f27b0a119720" containerName="istio-proxy"
Oct 30 20:56:20.433926 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.433894    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99069007-e0f8-4092-8fd6-f27b0a119720" containerName="istio-proxy"
Oct 30 20:56:20.434260 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.434052    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="99069007-e0f8-4092-8fd6-f27b0a119720" containerName="istio-proxy"
Oct 30 20:56:20.434260 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.434065    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="99069007-e0f8-4092-8fd6-f27b0a119720" containerName="istio-proxy"
Oct 30 20:56:20.434260 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.434084    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b4b151c-97ec-4a24-9569-53eba1c22d79" containerName="payload-pre-processing"
Oct 30 20:56:20.434260 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.434095    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d3048f3-fbc4-4718-ae16-55b2c6c54890" containerName="payload-processing"
Oct 30 20:56:20.437243 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.437224    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:56:20.496376 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.496340    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-9e0fb1-openshift-default-dockercfg-rnr68\""
Oct 30 20:56:20.520497 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.520456    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-credential-socket\") pod \"e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz\" (UID: \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\") " pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:56:20.520681 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.520516    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-istio-data\") pod \"e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz\" (UID: \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\") " pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:56:20.520681 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.520601    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-istiod-ca-cert\") pod \"e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz\" (UID: \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\") " pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:56:20.520813 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.520769    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-workload-socket\") pod \"e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz\" (UID: \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\") " pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:56:20.520813 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.520894    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-istio-envoy\") pod \"e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz\" (UID: \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\") " pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:56:20.521072 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.520939    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-workload-certs\") pod \"e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz\" (UID: \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\") " pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:56:20.521072 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.521013    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq97w\" (UniqueName: \"kubernetes.io/projected/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-kube-api-access-kq97w\") pod \"e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz\" (UID: \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\") " pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:56:20.521072 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.521047    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-istio-token\") pod \"e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz\" (UID: \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\") " pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:56:20.521247 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.521124    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-istio-podinfo\") pod \"e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz\" (UID: \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\") " pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:56:20.521304 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.521251    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkz4s\" (UniqueName: \"kubernetes.io/projected/9b4b151c-97ec-4a24-9569-53eba1c22d79-kube-api-access-jkz4s\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:20.521304 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.521268    5071 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b4b151c-97ec-4a24-9569-53eba1c22d79-plugins-config-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:20.544370 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.544340    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"]
Oct 30 20:56:20.557665 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.557627    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:20.622429 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.622394    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq97w\" (UniqueName: \"kubernetes.io/projected/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-kube-api-access-kq97w\") pod \"e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz\" (UID: \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\") " pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:56:20.622593 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.622435    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-istio-token\") pod \"e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz\" (UID: \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\") " pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:56:20.622593 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.622474    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-istio-podinfo\") pod \"e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz\" (UID: \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\") " pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:56:20.622593 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.622527    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-credential-socket\") pod \"e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz\" (UID: \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\") " pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:56:20.622781 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.622664    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-istio-data\") pod \"e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz\" (UID: \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\") " pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:56:20.622781 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.622710    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-istiod-ca-cert\") pod \"e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz\" (UID: \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\") " pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:56:20.622911 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.622811    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-workload-socket\") pod \"e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz\" (UID: \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\") " pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:56:20.622911 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.622857    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-credential-socket\") pod \"e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz\" (UID: \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\") " pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:56:20.623025 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.622927    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-istio-envoy\") pod \"e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz\" (UID: \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\") " pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:56:20.623025 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.622976    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-workload-certs\") pod \"e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz\" (UID: \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\") " pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:56:20.623120 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.623044    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-istio-data\") pod \"e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz\" (UID: \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\") " pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:56:20.623159 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.623115    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-workload-socket\") pod \"e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz\" (UID: \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\") " pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:56:20.623274 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.623256    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-workload-certs\") pod \"e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz\" (UID: \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\") " pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:56:20.623403 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.623383    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-istiod-ca-cert\") pod \"e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz\" (UID: \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\") " pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:56:20.625211 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.625188    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-istio-envoy\") pod \"e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz\" (UID: \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\") " pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:56:20.625211 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.625209    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-istio-podinfo\") pod \"e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz\" (UID: \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\") " pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:56:20.702024 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.701936    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq97w\" (UniqueName: \"kubernetes.io/projected/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-kube-api-access-kq97w\") pod \"e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz\" (UID: \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\") " pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:56:20.707737 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.707519    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-istio-token\") pod \"e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz\" (UID: \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\") " pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:56:20.739341 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.739302    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:20.754130 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:20.754102    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:56:21.043001 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:21.042976    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b86743-7c48bdc49b-xdsk6"
Oct 30 20:56:21.043453 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:21.042997    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b86743-7c48bdc49b-xdsk6" event={"ID":"4d3048f3-fbc4-4718-ae16-55b2c6c54890","Type":"ContainerDied","Data":"5aa8799fa8765b98a8453f1d278ee88c04a707738132565cdd8e5caea34173a1"}
Oct 30 20:56:21.043453 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:21.043103    5071 scope.go:122] "RemoveContainer" containerID="b1660487fa13d855e4da0aa4d9be71b57ac9154452d88d9a939bb15b5a0712c1"
Oct 30 20:56:21.044554 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:21.044524    5071 generic.go:356] "Generic (PLEG): container finished" podID="9b4b151c-97ec-4a24-9569-53eba1c22d79" containerID="fd9fddd9a7fba0ded36c2012a5bf760c4a96a03ffa040aa4c05ad5d2d4e5575e" exitCode=0
Oct 30 20:56:21.044641 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:21.044573    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b86743-7fbb475c6c-gfq7v" event={"ID":"9b4b151c-97ec-4a24-9569-53eba1c22d79","Type":"ContainerDied","Data":"fd9fddd9a7fba0ded36c2012a5bf760c4a96a03ffa040aa4c05ad5d2d4e5575e"}
Oct 30 20:56:21.044684 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:21.044658    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b86743-7fbb475c6c-gfq7v" event={"ID":"9b4b151c-97ec-4a24-9569-53eba1c22d79","Type":"ContainerDied","Data":"da3bbb0a957d217e3803d4966f7f72aaab060fbaa5f6c018890ece7bb812a73b"}
Oct 30 20:56:21.044684 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:21.044588    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b86743-7fbb475c6c-gfq7v"
Oct 30 20:56:21.054009 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:21.053991    5071 scope.go:122] "RemoveContainer" containerID="fd9fddd9a7fba0ded36c2012a5bf760c4a96a03ffa040aa4c05ad5d2d4e5575e"
Oct 30 20:56:21.063725 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:21.063705    5071 scope.go:122] "RemoveContainer" containerID="fd9fddd9a7fba0ded36c2012a5bf760c4a96a03ffa040aa4c05ad5d2d4e5575e"
Oct 30 20:56:21.064002 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:56:21.063975    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd9fddd9a7fba0ded36c2012a5bf760c4a96a03ffa040aa4c05ad5d2d4e5575e\": container with ID starting with fd9fddd9a7fba0ded36c2012a5bf760c4a96a03ffa040aa4c05ad5d2d4e5575e not found: ID does not exist" containerID="fd9fddd9a7fba0ded36c2012a5bf760c4a96a03ffa040aa4c05ad5d2d4e5575e"
Oct 30 20:56:21.064092 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:21.064012    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd9fddd9a7fba0ded36c2012a5bf760c4a96a03ffa040aa4c05ad5d2d4e5575e"} err="failed to get container status \"fd9fddd9a7fba0ded36c2012a5bf760c4a96a03ffa040aa4c05ad5d2d4e5575e\": rpc error: code = NotFound desc = could not find container \"fd9fddd9a7fba0ded36c2012a5bf760c4a96a03ffa040aa4c05ad5d2d4e5575e\": container with ID starting with fd9fddd9a7fba0ded36c2012a5bf760c4a96a03ffa040aa4c05ad5d2d4e5575e not found: ID does not exist"
Oct 30 20:56:21.189724 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:21.189691    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"]
Oct 30 20:56:21.189893 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:56:21.189700    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e3d8feb_58a2_40e0_b5b1_94acb06fe606.slice/crio-197ea65c44c3ebd32982ca242194a03548f7367bdb40fdb781af0aa6a4a3eb00 WatchSource:0}: Error finding container 197ea65c44c3ebd32982ca242194a03548f7367bdb40fdb781af0aa6a4a3eb00: Status 404 returned error can't find the container with id 197ea65c44c3ebd32982ca242194a03548f7367bdb40fdb781af0aa6a4a3eb00
Oct 30 20:56:21.192659 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:21.192622    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:56:21.192752 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:21.192693    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:56:21.192752 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:21.192724    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:56:21.212152 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:21.212128    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b86743-7fbb475c6c-gfq7v"]
Oct 30 20:56:21.218686 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:21.218660    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b86743-7fbb475c6c-gfq7v"]
Oct 30 20:56:21.327383 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:21.327291    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b86743-7c48bdc49b-xdsk6"]
Oct 30 20:56:21.336145 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:21.336115    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b86743-7c48bdc49b-xdsk6"]
Oct 30 20:56:22.054353 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:22.054321    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz" event={"ID":"7e3d8feb-58a2-40e0-b5b1-94acb06fe606","Type":"ContainerStarted","Data":"c79a9635fc675b303fb30614e9496e952b713e415fe8b6b259aa9c279c7f1c2e"}
Oct 30 20:56:22.054879 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:22.054371    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz" event={"ID":"7e3d8feb-58a2-40e0-b5b1-94acb06fe606","Type":"ContainerStarted","Data":"197ea65c44c3ebd32982ca242194a03548f7367bdb40fdb781af0aa6a4a3eb00"}
Oct 30 20:56:22.173341 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:22.173287    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz" podStartSLOduration=2.17327388 podStartE2EDuration="2.17327388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:22.171526806 +0000 UTC m=+1794.088548342" watchObservedRunningTime="2026-10-30 20:56:22.17327388 +0000 UTC m=+1794.090295417"
Oct 30 20:56:22.203420 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:22.203383    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d3048f3-fbc4-4718-ae16-55b2c6c54890" path="/var/lib/kubelet/pods/4d3048f3-fbc4-4718-ae16-55b2c6c54890/volumes"
Oct 30 20:56:22.203914 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:22.203893    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b4b151c-97ec-4a24-9569-53eba1c22d79" path="/var/lib/kubelet/pods/9b4b151c-97ec-4a24-9569-53eba1c22d79/volumes"
Oct 30 20:56:22.755214 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:22.755181    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:56:22.761501 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:22.761474    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:56:23.064807 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:23.064770    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:56:23.065783 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:23.065755    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:56:24.312274 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:24.312242    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:24.971657 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:24.971618    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:25.197350 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:25.197319    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:25.399472 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:25.399432    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:28.263370 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:28.263336    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:28.513250 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:28.495736    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:28.666145 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:28.666119    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-9e0fb1-78475c7fd7-gkjhq"]
Oct 30 20:56:28.670756 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:28.670730    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-9e0fb1-78475c7fd7-gkjhq"
Oct 30 20:56:28.701670 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:28.701644    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8qvb_91fe9e06-0c30-4000-bddb-2ba89cd048cc/console-operator/2.log"
Oct 30 20:56:28.710081 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:28.710053    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q4f4h_80cae79e-b60f-4efe-bb58-e77753a457e5/ovn-acl-logging/0.log"
Oct 30 20:56:28.716239 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:28.716219    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8qvb_91fe9e06-0c30-4000-bddb-2ba89cd048cc/console-operator/2.log"
Oct 30 20:56:28.727653 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:28.727624    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q4f4h_80cae79e-b60f-4efe-bb58-e77753a457e5/ovn-acl-logging/0.log"
Oct 30 20:56:28.728080 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:28.728060    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-9e0fb1-78475c7fd7-gkjhq"]
Oct 30 20:56:28.775901 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:28.774180    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-9e0fb1-dockercfg-6cp4s\""
Oct 30 20:56:28.775901 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:28.774313    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-9e0fb1\""
Oct 30 20:56:29.037766 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:29.037666    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm7k4\" (UniqueName: \"kubernetes.io/projected/dc24a7fe-b98b-4e31-ac64-3b80da898040-kube-api-access-jm7k4\") pod \"payload-pre-processing-e2e-shared-b-w6-9e0fb1-78475c7fd7-gkjhq\" (UID: \"dc24a7fe-b98b-4e31-ac64-3b80da898040\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-9e0fb1-78475c7fd7-gkjhq"
Oct 30 20:56:29.037985 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:29.037963    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc24a7fe-b98b-4e31-ac64-3b80da898040-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-9e0fb1-78475c7fd7-gkjhq\" (UID: \"dc24a7fe-b98b-4e31-ac64-3b80da898040\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-9e0fb1-78475c7fd7-gkjhq"
Oct 30 20:56:29.039366 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:29.039337    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:29.041601 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:29.041580    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-9e0fb1-7f7c8cbcc6-xzvm7"]
Oct 30 20:56:29.054953 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:29.054933    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-9e0fb1-7f7c8cbcc6-xzvm7"]
Oct 30 20:56:29.055238 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:29.055224    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-9e0fb1-7f7c8cbcc6-xzvm7"
Oct 30 20:56:29.147729 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:29.144011    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm7k4\" (UniqueName: \"kubernetes.io/projected/dc24a7fe-b98b-4e31-ac64-3b80da898040-kube-api-access-jm7k4\") pod \"payload-pre-processing-e2e-shared-b-w6-9e0fb1-78475c7fd7-gkjhq\" (UID: \"dc24a7fe-b98b-4e31-ac64-3b80da898040\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-9e0fb1-78475c7fd7-gkjhq"
Oct 30 20:56:29.147729 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:29.144093    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qf9b\" (UniqueName: \"kubernetes.io/projected/d6cafaf5-279b-4ec5-b830-b27a7c00eb19-kube-api-access-4qf9b\") pod \"payload-processing-e2e-shared-b-w6-9e0fb1-7f7c8cbcc6-xzvm7\" (UID: \"d6cafaf5-279b-4ec5-b830-b27a7c00eb19\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-9e0fb1-7f7c8cbcc6-xzvm7"
Oct 30 20:56:29.147729 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:29.144209    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc24a7fe-b98b-4e31-ac64-3b80da898040-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-9e0fb1-78475c7fd7-gkjhq\" (UID: \"dc24a7fe-b98b-4e31-ac64-3b80da898040\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-9e0fb1-78475c7fd7-gkjhq"
Oct 30 20:56:29.147729 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:29.144285    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6cafaf5-279b-4ec5-b830-b27a7c00eb19-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-9e0fb1-7f7c8cbcc6-xzvm7\" (UID: \"d6cafaf5-279b-4ec5-b830-b27a7c00eb19\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-9e0fb1-7f7c8cbcc6-xzvm7"
Oct 30 20:56:29.147729 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:29.145723    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc24a7fe-b98b-4e31-ac64-3b80da898040-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-9e0fb1-78475c7fd7-gkjhq\" (UID: \"dc24a7fe-b98b-4e31-ac64-3b80da898040\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-9e0fb1-78475c7fd7-gkjhq"
Oct 30 20:56:29.203676 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:29.203629    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm7k4\" (UniqueName: \"kubernetes.io/projected/dc24a7fe-b98b-4e31-ac64-3b80da898040-kube-api-access-jm7k4\") pod \"payload-pre-processing-e2e-shared-b-w6-9e0fb1-78475c7fd7-gkjhq\" (UID: \"dc24a7fe-b98b-4e31-ac64-3b80da898040\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-9e0fb1-78475c7fd7-gkjhq"
Oct 30 20:56:29.245509 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:29.245466    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qf9b\" (UniqueName: \"kubernetes.io/projected/d6cafaf5-279b-4ec5-b830-b27a7c00eb19-kube-api-access-4qf9b\") pod \"payload-processing-e2e-shared-b-w6-9e0fb1-7f7c8cbcc6-xzvm7\" (UID: \"d6cafaf5-279b-4ec5-b830-b27a7c00eb19\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-9e0fb1-7f7c8cbcc6-xzvm7"
Oct 30 20:56:29.245682 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:29.245668    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6cafaf5-279b-4ec5-b830-b27a7c00eb19-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-9e0fb1-7f7c8cbcc6-xzvm7\" (UID: \"d6cafaf5-279b-4ec5-b830-b27a7c00eb19\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-9e0fb1-7f7c8cbcc6-xzvm7"
Oct 30 20:56:29.246437 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:29.246415    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6cafaf5-279b-4ec5-b830-b27a7c00eb19-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-9e0fb1-7f7c8cbcc6-xzvm7\" (UID: \"d6cafaf5-279b-4ec5-b830-b27a7c00eb19\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-9e0fb1-7f7c8cbcc6-xzvm7"
Oct 30 20:56:29.288426 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:29.288360    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-9e0fb1-78475c7fd7-gkjhq"
Oct 30 20:56:29.306156 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:29.306131    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qf9b\" (UniqueName: \"kubernetes.io/projected/d6cafaf5-279b-4ec5-b830-b27a7c00eb19-kube-api-access-4qf9b\") pod \"payload-processing-e2e-shared-b-w6-9e0fb1-7f7c8cbcc6-xzvm7\" (UID: \"d6cafaf5-279b-4ec5-b830-b27a7c00eb19\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-9e0fb1-7f7c8cbcc6-xzvm7"
Oct 30 20:56:29.469897 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:29.465902    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-9e0fb1-7f7c8cbcc6-xzvm7"
Oct 30 20:56:29.811634 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:29.811604    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-9e0fb1-78475c7fd7-gkjhq"]
Oct 30 20:56:29.812263 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:56:29.812193    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc24a7fe_b98b_4e31_ac64_3b80da898040.slice/crio-0acc7392e26db3394c943628935670a514fd3933b5415b99411111bd1858c956 WatchSource:0}: Error finding container 0acc7392e26db3394c943628935670a514fd3933b5415b99411111bd1858c956: Status 404 returned error can't find the container with id 0acc7392e26db3394c943628935670a514fd3933b5415b99411111bd1858c956
Oct 30 20:56:29.911913 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:29.911890    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-9e0fb1-7f7c8cbcc6-xzvm7"]
Oct 30 20:56:29.912366 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:56:29.912345    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6cafaf5_279b_4ec5_b830_b27a7c00eb19.slice/crio-92d75f355392807b9fcf28fd29ec5ee38d3707a5a8893db8dea824a8cd69bd2a WatchSource:0}: Error finding container 92d75f355392807b9fcf28fd29ec5ee38d3707a5a8893db8dea824a8cd69bd2a: Status 404 returned error can't find the container with id 92d75f355392807b9fcf28fd29ec5ee38d3707a5a8893db8dea824a8cd69bd2a
Oct 30 20:56:30.191882 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:30.190611    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-9e0fb1-7f7c8cbcc6-xzvm7" event={"ID":"d6cafaf5-279b-4ec5-b830-b27a7c00eb19","Type":"ContainerStarted","Data":"bdcd52dbb88e058054665f444178954745819b593e30fb45fcdc813685bd7346"}
Oct 30 20:56:30.191882 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:30.190689    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-9e0fb1-7f7c8cbcc6-xzvm7" event={"ID":"d6cafaf5-279b-4ec5-b830-b27a7c00eb19","Type":"ContainerStarted","Data":"92d75f355392807b9fcf28fd29ec5ee38d3707a5a8893db8dea824a8cd69bd2a"}
Oct 30 20:56:30.191882 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:30.191751    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-9e0fb1-7f7c8cbcc6-xzvm7"
Oct 30 20:56:30.235700 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:30.235670    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-9e0fb1-78475c7fd7-gkjhq"
Oct 30 20:56:30.235958 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:30.235933    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-9e0fb1-78475c7fd7-gkjhq" event={"ID":"dc24a7fe-b98b-4e31-ac64-3b80da898040","Type":"ContainerStarted","Data":"1b6642ae87b59e8a97266a4e379238b11b4563a49b0f5dde44127b399129a45b"}
Oct 30 20:56:30.247588 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:30.247559    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-9e0fb1-78475c7fd7-gkjhq" event={"ID":"dc24a7fe-b98b-4e31-ac64-3b80da898040","Type":"ContainerStarted","Data":"0acc7392e26db3394c943628935670a514fd3933b5415b99411111bd1858c956"}
Oct 30 20:56:30.309391 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:30.309283    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-9e0fb1-7f7c8cbcc6-xzvm7" podStartSLOduration=2.309270484 podStartE2EDuration="2.309270484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:30.309137372 +0000 UTC m=+1802.226158908" watchObservedRunningTime="2026-10-30 20:56:30.309270484 +0000 UTC m=+1802.226292020"
Oct 30 20:56:30.497896 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:30.491838    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-9e0fb1-78475c7fd7-gkjhq" podStartSLOduration=2.491799651 podStartE2EDuration="2.491799651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:30.4312537 +0000 UTC m=+1802.348275239" watchObservedRunningTime="2026-10-30 20:56:30.491799651 +0000 UTC m=+1802.408821188"
Oct 30 20:56:30.887709 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:30.887675    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:31.179460 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:31.179364    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:31.348161 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:31.348103    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:31.878803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:31.878773    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:32.796894 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:32.796860    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:33.353026 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:33.352998    5071 scope.go:122] "RemoveContainer" containerID="ce46d77cd42b8fdeb83e37a9b9001432751063d511835f65e58f6bd905bc0cf8"
Oct 30 20:56:33.364942 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:33.364919    5071 scope.go:122] "RemoveContainer" containerID="aa56c88aa32cb42c8face068e993bc20f28cf8122db9adbe39a3fe0a18eae22d"
Oct 30 20:56:34.312905 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:34.312854    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:36.670631 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:36.670597    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:38.545362 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:38.545323    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:38.768049 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:38.768009    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:38.886277 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:38.886190    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"]
Oct 30 20:56:38.886529 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:38.886496    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk" podUID="26c59b24-19e6-4911-a608-5cefe1ab8204" containerName="istio-proxy" containerID="cri-o://4a459da3a7bbe34a743f24736edce169c166367cd0f57ef85e1985652a3f14a9" gracePeriod=30
Oct 30 20:56:38.921848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:38.921789    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"]
Oct 30 20:56:38.922049 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:38.922010    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts" podUID="7299979e-94b4-4d26-a860-c88b6136faf2" containerName="istio-proxy" containerID="cri-o://78f1719a501680ab87a17a2667dbae3db84a6f30a1c9aa976c4a04e878861d1a" gracePeriod=30
Oct 30 20:56:38.933549 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:38.933507    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:42.254102 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:42.254070    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-9e0fb1-7f7c8cbcc6-xzvm7"
Oct 30 20:56:42.254622 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:42.254149    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-9e0fb1-78475c7fd7-gkjhq"
Oct 30 20:56:42.264995 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:42.264963    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:44.241004 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.240974    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"
Oct 30 20:56:44.270513 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.270486    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"
Oct 30 20:56:44.390035 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.389998    5071 generic.go:356] "Generic (PLEG): container finished" podID="26c59b24-19e6-4911-a608-5cefe1ab8204" containerID="4a459da3a7bbe34a743f24736edce169c166367cd0f57ef85e1985652a3f14a9" exitCode=0
Oct 30 20:56:44.390428 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.390392    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk" event={"ID":"26c59b24-19e6-4911-a608-5cefe1ab8204","Type":"ContainerDied","Data":"4a459da3a7bbe34a743f24736edce169c166367cd0f57ef85e1985652a3f14a9"}
Oct 30 20:56:44.390552 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.390462    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk" event={"ID":"26c59b24-19e6-4911-a608-5cefe1ab8204","Type":"ContainerDied","Data":"97866626ca1da893026e1346afda81b91867066f0ddf48cc2d3b8c5f8a837860"}
Oct 30 20:56:44.390552 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.390496    5071 scope.go:122] "RemoveContainer" containerID="4a459da3a7bbe34a743f24736edce169c166367cd0f57ef85e1985652a3f14a9"
Oct 30 20:56:44.390705 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.390683    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"
Oct 30 20:56:44.393259 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.393230    5071 generic.go:356] "Generic (PLEG): container finished" podID="7299979e-94b4-4d26-a860-c88b6136faf2" containerID="78f1719a501680ab87a17a2667dbae3db84a6f30a1c9aa976c4a04e878861d1a" exitCode=0
Oct 30 20:56:44.393455 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.393433    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"
Oct 30 20:56:44.393699 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.393278    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts" event={"ID":"7299979e-94b4-4d26-a860-c88b6136faf2","Type":"ContainerDied","Data":"78f1719a501680ab87a17a2667dbae3db84a6f30a1c9aa976c4a04e878861d1a"}
Oct 30 20:56:44.393795 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.393736    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts" event={"ID":"7299979e-94b4-4d26-a860-c88b6136faf2","Type":"ContainerDied","Data":"cebc659d2daf172e099bc338d4f74a23f1d68bdafcf7da4364202b5da3afbd9b"}
Oct 30 20:56:44.403636 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.403618    5071 scope.go:122] "RemoveContainer" containerID="4a459da3a7bbe34a743f24736edce169c166367cd0f57ef85e1985652a3f14a9"
Oct 30 20:56:44.404000 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:56:44.403974    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a459da3a7bbe34a743f24736edce169c166367cd0f57ef85e1985652a3f14a9\": container with ID starting with 4a459da3a7bbe34a743f24736edce169c166367cd0f57ef85e1985652a3f14a9 not found: ID does not exist" containerID="4a459da3a7bbe34a743f24736edce169c166367cd0f57ef85e1985652a3f14a9"
Oct 30 20:56:44.404073 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.404007    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a459da3a7bbe34a743f24736edce169c166367cd0f57ef85e1985652a3f14a9"} err="failed to get container status \"4a459da3a7bbe34a743f24736edce169c166367cd0f57ef85e1985652a3f14a9\": rpc error: code = NotFound desc = could not find container \"4a459da3a7bbe34a743f24736edce169c166367cd0f57ef85e1985652a3f14a9\": container with ID starting with 4a459da3a7bbe34a743f24736edce169c166367cd0f57ef85e1985652a3f14a9 not found: ID does not exist"
Oct 30 20:56:44.404073 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.404024    5071 scope.go:122] "RemoveContainer" containerID="78f1719a501680ab87a17a2667dbae3db84a6f30a1c9aa976c4a04e878861d1a"
Oct 30 20:56:44.416905 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.416885    5071 scope.go:122] "RemoveContainer" containerID="78f1719a501680ab87a17a2667dbae3db84a6f30a1c9aa976c4a04e878861d1a"
Oct 30 20:56:44.417281 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:56:44.417259    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78f1719a501680ab87a17a2667dbae3db84a6f30a1c9aa976c4a04e878861d1a\": container with ID starting with 78f1719a501680ab87a17a2667dbae3db84a6f30a1c9aa976c4a04e878861d1a not found: ID does not exist" containerID="78f1719a501680ab87a17a2667dbae3db84a6f30a1c9aa976c4a04e878861d1a"
Oct 30 20:56:44.417368 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.417289    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78f1719a501680ab87a17a2667dbae3db84a6f30a1c9aa976c4a04e878861d1a"} err="failed to get container status \"78f1719a501680ab87a17a2667dbae3db84a6f30a1c9aa976c4a04e878861d1a\": rpc error: code = NotFound desc = could not find container \"78f1719a501680ab87a17a2667dbae3db84a6f30a1c9aa976c4a04e878861d1a\": container with ID starting with 78f1719a501680ab87a17a2667dbae3db84a6f30a1c9aa976c4a04e878861d1a not found: ID does not exist"
Oct 30 20:56:44.420796 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.420769    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26c59b24-19e6-4911-a608-5cefe1ab8204-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c59b24-19e6-4911-a608-5cefe1ab8204-credential-socket\") pod \"26c59b24-19e6-4911-a608-5cefe1ab8204\" (UID: \"26c59b24-19e6-4911-a608-5cefe1ab8204\") "
Oct 30 20:56:44.420939 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.420870    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26c59b24-19e6-4911-a608-5cefe1ab8204-kube-api-access-mvlj4\" (UniqueName: \"kubernetes.io/projected/26c59b24-19e6-4911-a608-5cefe1ab8204-kube-api-access-mvlj4\") pod \"26c59b24-19e6-4911-a608-5cefe1ab8204\" (UID: \"26c59b24-19e6-4911-a608-5cefe1ab8204\") "
Oct 30 20:56:44.420939 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.420908    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26c59b24-19e6-4911-a608-5cefe1ab8204-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c59b24-19e6-4911-a608-5cefe1ab8204-workload-socket\") pod \"26c59b24-19e6-4911-a608-5cefe1ab8204\" (UID: \"26c59b24-19e6-4911-a608-5cefe1ab8204\") "
Oct 30 20:56:44.421065 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.420981    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7299979e-94b4-4d26-a860-c88b6136faf2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7299979e-94b4-4d26-a860-c88b6136faf2-workload-socket\") pod \"7299979e-94b4-4d26-a860-c88b6136faf2\" (UID: \"7299979e-94b4-4d26-a860-c88b6136faf2\") "
Oct 30 20:56:44.421065 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.421016    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7299979e-94b4-4d26-a860-c88b6136faf2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7299979e-94b4-4d26-a860-c88b6136faf2-istio-envoy\") pod \"7299979e-94b4-4d26-a860-c88b6136faf2\" (UID: \"7299979e-94b4-4d26-a860-c88b6136faf2\") "
Oct 30 20:56:44.421065 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.421049    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26c59b24-19e6-4911-a608-5cefe1ab8204-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26c59b24-19e6-4911-a608-5cefe1ab8204-istio-data\") pod \"26c59b24-19e6-4911-a608-5cefe1ab8204\" (UID: \"26c59b24-19e6-4911-a608-5cefe1ab8204\") "
Oct 30 20:56:44.421217 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.421086    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7299979e-94b4-4d26-a860-c88b6136faf2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7299979e-94b4-4d26-a860-c88b6136faf2-credential-socket\") pod \"7299979e-94b4-4d26-a860-c88b6136faf2\" (UID: \"7299979e-94b4-4d26-a860-c88b6136faf2\") "
Oct 30 20:56:44.421217 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.421129    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7299979e-94b4-4d26-a860-c88b6136faf2-kube-api-access-2zj6b\" (UniqueName: \"kubernetes.io/projected/7299979e-94b4-4d26-a860-c88b6136faf2-kube-api-access-2zj6b\") pod \"7299979e-94b4-4d26-a860-c88b6136faf2\" (UID: \"7299979e-94b4-4d26-a860-c88b6136faf2\") "
Oct 30 20:56:44.421217 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.421167    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26c59b24-19e6-4911-a608-5cefe1ab8204-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26c59b24-19e6-4911-a608-5cefe1ab8204-istiod-ca-cert\") pod \"26c59b24-19e6-4911-a608-5cefe1ab8204\" (UID: \"26c59b24-19e6-4911-a608-5cefe1ab8204\") "
Oct 30 20:56:44.421380 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.421281    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7299979e-94b4-4d26-a860-c88b6136faf2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7299979e-94b4-4d26-a860-c88b6136faf2-workload-certs\") pod \"7299979e-94b4-4d26-a860-c88b6136faf2\" (UID: \"7299979e-94b4-4d26-a860-c88b6136faf2\") "
Oct 30 20:56:44.421380 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.421321    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7299979e-94b4-4d26-a860-c88b6136faf2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7299979e-94b4-4d26-a860-c88b6136faf2-istio-data\") pod \"7299979e-94b4-4d26-a860-c88b6136faf2\" (UID: \"7299979e-94b4-4d26-a860-c88b6136faf2\") "
Oct 30 20:56:44.421380 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.421351    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26c59b24-19e6-4911-a608-5cefe1ab8204-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26c59b24-19e6-4911-a608-5cefe1ab8204-workload-certs\") pod \"26c59b24-19e6-4911-a608-5cefe1ab8204\" (UID: \"26c59b24-19e6-4911-a608-5cefe1ab8204\") "
Oct 30 20:56:44.421572 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.421390    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26c59b24-19e6-4911-a608-5cefe1ab8204-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26c59b24-19e6-4911-a608-5cefe1ab8204-istio-envoy\") pod \"26c59b24-19e6-4911-a608-5cefe1ab8204\" (UID: \"26c59b24-19e6-4911-a608-5cefe1ab8204\") "
Oct 30 20:56:44.421572 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.421413    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26c59b24-19e6-4911-a608-5cefe1ab8204-istio-token\" (UniqueName: \"kubernetes.io/projected/26c59b24-19e6-4911-a608-5cefe1ab8204-istio-token\") pod \"26c59b24-19e6-4911-a608-5cefe1ab8204\" (UID: \"26c59b24-19e6-4911-a608-5cefe1ab8204\") "
Oct 30 20:56:44.421572 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.421504    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7299979e-94b4-4d26-a860-c88b6136faf2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7299979e-94b4-4d26-a860-c88b6136faf2-istio-podinfo\") pod \"7299979e-94b4-4d26-a860-c88b6136faf2\" (UID: \"7299979e-94b4-4d26-a860-c88b6136faf2\") "
Oct 30 20:56:44.421572 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.421540    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7299979e-94b4-4d26-a860-c88b6136faf2-istio-token\" (UniqueName: \"kubernetes.io/projected/7299979e-94b4-4d26-a860-c88b6136faf2-istio-token\") pod \"7299979e-94b4-4d26-a860-c88b6136faf2\" (UID: \"7299979e-94b4-4d26-a860-c88b6136faf2\") "
Oct 30 20:56:44.421791 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.421588    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7299979e-94b4-4d26-a860-c88b6136faf2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7299979e-94b4-4d26-a860-c88b6136faf2-istiod-ca-cert\") pod \"7299979e-94b4-4d26-a860-c88b6136faf2\" (UID: \"7299979e-94b4-4d26-a860-c88b6136faf2\") "
Oct 30 20:56:44.421791 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.421648    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26c59b24-19e6-4911-a608-5cefe1ab8204-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26c59b24-19e6-4911-a608-5cefe1ab8204-istio-podinfo\") pod \"26c59b24-19e6-4911-a608-5cefe1ab8204\" (UID: \"26c59b24-19e6-4911-a608-5cefe1ab8204\") "
Oct 30 20:56:44.422295 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.422271    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26c59b24-19e6-4911-a608-5cefe1ab8204-istiod-ca-cert" pod "26c59b24-19e6-4911-a608-5cefe1ab8204" (UID: "26c59b24-19e6-4911-a608-5cefe1ab8204"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:56:44.422710 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.422687    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7299979e-94b4-4d26-a860-c88b6136faf2-credential-socket" pod "7299979e-94b4-4d26-a860-c88b6136faf2" (UID: "7299979e-94b4-4d26-a860-c88b6136faf2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:56:44.423660 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.422958    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c59b24-19e6-4911-a608-5cefe1ab8204-istio-data" pod "26c59b24-19e6-4911-a608-5cefe1ab8204" (UID: "26c59b24-19e6-4911-a608-5cefe1ab8204"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:56:44.423660 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.423268    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c59b24-19e6-4911-a608-5cefe1ab8204-workload-socket" pod "26c59b24-19e6-4911-a608-5cefe1ab8204" (UID: "26c59b24-19e6-4911-a608-5cefe1ab8204"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:56:44.424972 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.424929    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c59b24-19e6-4911-a608-5cefe1ab8204-workload-certs" pod "26c59b24-19e6-4911-a608-5cefe1ab8204" (UID: "26c59b24-19e6-4911-a608-5cefe1ab8204"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:56:44.425221 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.425198    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c59b24-19e6-4911-a608-5cefe1ab8204-credential-socket" pod "26c59b24-19e6-4911-a608-5cefe1ab8204" (UID: "26c59b24-19e6-4911-a608-5cefe1ab8204"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:56:44.425805 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.425705    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7299979e-94b4-4d26-a860-c88b6136faf2-workload-certs" pod "7299979e-94b4-4d26-a860-c88b6136faf2" (UID: "7299979e-94b4-4d26-a860-c88b6136faf2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:56:44.425805 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.425780    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7299979e-94b4-4d26-a860-c88b6136faf2-istio-data" pod "7299979e-94b4-4d26-a860-c88b6136faf2" (UID: "7299979e-94b4-4d26-a860-c88b6136faf2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:56:44.426005 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.425897    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7299979e-94b4-4d26-a860-c88b6136faf2-istiod-ca-cert" pod "7299979e-94b4-4d26-a860-c88b6136faf2" (UID: "7299979e-94b4-4d26-a860-c88b6136faf2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:56:44.426203 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.426181    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7299979e-94b4-4d26-a860-c88b6136faf2-workload-socket" pod "7299979e-94b4-4d26-a860-c88b6136faf2" (UID: "7299979e-94b4-4d26-a860-c88b6136faf2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:56:44.433108 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.433086    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26c59b24-19e6-4911-a608-5cefe1ab8204-istio-podinfo" pod "26c59b24-19e6-4911-a608-5cefe1ab8204" (UID: "26c59b24-19e6-4911-a608-5cefe1ab8204"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:56:44.433910 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.433883    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7299979e-94b4-4d26-a860-c88b6136faf2-istio-podinfo" pod "7299979e-94b4-4d26-a860-c88b6136faf2" (UID: "7299979e-94b4-4d26-a860-c88b6136faf2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:56:44.433995 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.433976    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26c59b24-19e6-4911-a608-5cefe1ab8204-kube-api-access-mvlj4" pod "26c59b24-19e6-4911-a608-5cefe1ab8204" (UID: "26c59b24-19e6-4911-a608-5cefe1ab8204"). InnerVolumeSpecName "kube-api-access-mvlj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:56:44.434058 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.433995    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7299979e-94b4-4d26-a860-c88b6136faf2-kube-api-access-2zj6b" pod "7299979e-94b4-4d26-a860-c88b6136faf2" (UID: "7299979e-94b4-4d26-a860-c88b6136faf2"). InnerVolumeSpecName "kube-api-access-2zj6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:56:44.434058 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.434005    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7299979e-94b4-4d26-a860-c88b6136faf2-istio-token" pod "7299979e-94b4-4d26-a860-c88b6136faf2" (UID: "7299979e-94b4-4d26-a860-c88b6136faf2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:56:44.434135 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.434081    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7299979e-94b4-4d26-a860-c88b6136faf2-istio-envoy" pod "7299979e-94b4-4d26-a860-c88b6136faf2" (UID: "7299979e-94b4-4d26-a860-c88b6136faf2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:56:44.434854 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.434789    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26c59b24-19e6-4911-a608-5cefe1ab8204-istio-token" pod "26c59b24-19e6-4911-a608-5cefe1ab8204" (UID: "26c59b24-19e6-4911-a608-5cefe1ab8204"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:56:44.435667 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.435641    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c59b24-19e6-4911-a608-5cefe1ab8204-istio-envoy" pod "26c59b24-19e6-4911-a608-5cefe1ab8204" (UID: "26c59b24-19e6-4911-a608-5cefe1ab8204"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:56:44.523793 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.523714    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7299979e-94b4-4d26-a860-c88b6136faf2-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:44.523793 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.523742    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7299979e-94b4-4d26-a860-c88b6136faf2-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:44.523793 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.523752    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7299979e-94b4-4d26-a860-c88b6136faf2-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:44.523793 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.523761    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26c59b24-19e6-4911-a608-5cefe1ab8204-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:44.523793 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.523769    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c59b24-19e6-4911-a608-5cefe1ab8204-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:44.523793 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.523778    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvlj4\" (UniqueName: \"kubernetes.io/projected/26c59b24-19e6-4911-a608-5cefe1ab8204-kube-api-access-mvlj4\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:44.523793 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.523788    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26c59b24-19e6-4911-a608-5cefe1ab8204-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:44.523793 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.523797    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7299979e-94b4-4d26-a860-c88b6136faf2-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:44.524199 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.523807    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7299979e-94b4-4d26-a860-c88b6136faf2-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:44.524199 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.523816    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26c59b24-19e6-4911-a608-5cefe1ab8204-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:44.524199 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.523867    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7299979e-94b4-4d26-a860-c88b6136faf2-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:44.524199 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.523881    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zj6b\" (UniqueName: \"kubernetes.io/projected/7299979e-94b4-4d26-a860-c88b6136faf2-kube-api-access-2zj6b\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:44.524199 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.523890    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26c59b24-19e6-4911-a608-5cefe1ab8204-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:44.524199 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.523904    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7299979e-94b4-4d26-a860-c88b6136faf2-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:44.524199 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.523915    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7299979e-94b4-4d26-a860-c88b6136faf2-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:44.524199 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.523924    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26c59b24-19e6-4911-a608-5cefe1ab8204-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:44.524199 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.523933    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26c59b24-19e6-4911-a608-5cefe1ab8204-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:44.524199 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.523941    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26c59b24-19e6-4911-a608-5cefe1ab8204-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:44.822388 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.822356    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"]
Oct 30 20:56:44.828465 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.828432    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-scjpk"]
Oct 30 20:56:44.940408 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.940379    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"]
Oct 30 20:56:44.945475 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:44.945447    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-86d620-openshift-default-6646b648bc-cxgts"]
Oct 30 20:56:46.202574 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:46.202542    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26c59b24-19e6-4911-a608-5cefe1ab8204" path="/var/lib/kubelet/pods/26c59b24-19e6-4911-a608-5cefe1ab8204/volumes"
Oct 30 20:56:46.203087 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:46.203072    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7299979e-94b4-4d26-a860-c88b6136faf2" path="/var/lib/kubelet/pods/7299979e-94b4-4d26-a860-c88b6136faf2/volumes"
Oct 30 20:56:46.826129 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:56:46.826036    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:56:47.657237 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:47.657191    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:49.043769 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:49.043380    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:49.423786 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:49.423700    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:50.526267 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.526225    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"]
Oct 30 20:56:50.527192 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.527168    5071 cpu_manager.go:414] "containerMap: removing container" podUID="26c59b24-19e6-4911-a608-5cefe1ab8204" containerName="istio-proxy"
Oct 30 20:56:50.527192 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.527191    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26c59b24-19e6-4911-a608-5cefe1ab8204" containerName="istio-proxy"
Oct 30 20:56:50.527387 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.527236    5071 cpu_manager.go:414] "containerMap: removing container" podUID="7299979e-94b4-4d26-a860-c88b6136faf2" containerName="istio-proxy"
Oct 30 20:56:50.527387 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.527244    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7299979e-94b4-4d26-a860-c88b6136faf2" containerName="istio-proxy"
Oct 30 20:56:50.527502 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.527431    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="26c59b24-19e6-4911-a608-5cefe1ab8204" containerName="istio-proxy"
Oct 30 20:56:50.527502 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.527450    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="7299979e-94b4-4d26-a860-c88b6136faf2" containerName="istio-proxy"
Oct 30 20:56:50.531066 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.531046    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 20:56:50.589549 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.589519    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-19176d-openshift-default-dockercfg-wsnj4\""
Oct 30 20:56:50.641912 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.641873    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:50.642100 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.641962    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e671c902-62e8-48fc-bf30-6a2505c31167-istiod-ca-cert\") pod \"e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4\" (UID: \"e671c902-62e8-48fc-bf30-6a2505c31167\") " pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 20:56:50.642100 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.642009    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e671c902-62e8-48fc-bf30-6a2505c31167-credential-socket\") pod \"e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4\" (UID: \"e671c902-62e8-48fc-bf30-6a2505c31167\") " pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 20:56:50.642550 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.642528    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"]
Oct 30 20:56:50.642798 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.642770    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e671c902-62e8-48fc-bf30-6a2505c31167-workload-socket\") pod \"e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4\" (UID: \"e671c902-62e8-48fc-bf30-6a2505c31167\") " pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 20:56:50.643251 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.642812    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e671c902-62e8-48fc-bf30-6a2505c31167-istio-podinfo\") pod \"e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4\" (UID: \"e671c902-62e8-48fc-bf30-6a2505c31167\") " pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 20:56:50.643251 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.642858    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e671c902-62e8-48fc-bf30-6a2505c31167-istio-token\") pod \"e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4\" (UID: \"e671c902-62e8-48fc-bf30-6a2505c31167\") " pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 20:56:50.643251 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.642936    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z6tw\" (UniqueName: \"kubernetes.io/projected/e671c902-62e8-48fc-bf30-6a2505c31167-kube-api-access-5z6tw\") pod \"e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4\" (UID: \"e671c902-62e8-48fc-bf30-6a2505c31167\") " pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 20:56:50.643251 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.643107    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e671c902-62e8-48fc-bf30-6a2505c31167-workload-certs\") pod \"e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4\" (UID: \"e671c902-62e8-48fc-bf30-6a2505c31167\") " pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 20:56:50.643251 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.643188    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e671c902-62e8-48fc-bf30-6a2505c31167-istio-envoy\") pod \"e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4\" (UID: \"e671c902-62e8-48fc-bf30-6a2505c31167\") " pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 20:56:50.643499 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.643301    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e671c902-62e8-48fc-bf30-6a2505c31167-istio-data\") pod \"e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4\" (UID: \"e671c902-62e8-48fc-bf30-6a2505c31167\") " pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 20:56:50.749775 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.745353    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e671c902-62e8-48fc-bf30-6a2505c31167-istio-envoy\") pod \"e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4\" (UID: \"e671c902-62e8-48fc-bf30-6a2505c31167\") " pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 20:56:50.749775 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.745458    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e671c902-62e8-48fc-bf30-6a2505c31167-istio-data\") pod \"e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4\" (UID: \"e671c902-62e8-48fc-bf30-6a2505c31167\") " pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 20:56:50.749775 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.745510    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e671c902-62e8-48fc-bf30-6a2505c31167-istiod-ca-cert\") pod \"e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4\" (UID: \"e671c902-62e8-48fc-bf30-6a2505c31167\") " pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 20:56:50.749775 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.745535    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e671c902-62e8-48fc-bf30-6a2505c31167-credential-socket\") pod \"e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4\" (UID: \"e671c902-62e8-48fc-bf30-6a2505c31167\") " pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 20:56:50.749775 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.745631    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e671c902-62e8-48fc-bf30-6a2505c31167-workload-socket\") pod \"e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4\" (UID: \"e671c902-62e8-48fc-bf30-6a2505c31167\") " pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 20:56:50.749775 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.745688    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e671c902-62e8-48fc-bf30-6a2505c31167-istio-podinfo\") pod \"e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4\" (UID: \"e671c902-62e8-48fc-bf30-6a2505c31167\") " pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 20:56:50.749775 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.745749    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e671c902-62e8-48fc-bf30-6a2505c31167-istio-token\") pod \"e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4\" (UID: \"e671c902-62e8-48fc-bf30-6a2505c31167\") " pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 20:56:50.749775 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.745843    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z6tw\" (UniqueName: \"kubernetes.io/projected/e671c902-62e8-48fc-bf30-6a2505c31167-kube-api-access-5z6tw\") pod \"e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4\" (UID: \"e671c902-62e8-48fc-bf30-6a2505c31167\") " pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 20:56:50.749775 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.745943    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e671c902-62e8-48fc-bf30-6a2505c31167-workload-certs\") pod \"e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4\" (UID: \"e671c902-62e8-48fc-bf30-6a2505c31167\") " pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 20:56:50.749775 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.746616    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e671c902-62e8-48fc-bf30-6a2505c31167-workload-certs\") pod \"e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4\" (UID: \"e671c902-62e8-48fc-bf30-6a2505c31167\") " pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 20:56:50.749775 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.747468    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e671c902-62e8-48fc-bf30-6a2505c31167-workload-socket\") pod \"e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4\" (UID: \"e671c902-62e8-48fc-bf30-6a2505c31167\") " pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 20:56:50.749775 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.747729    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e671c902-62e8-48fc-bf30-6a2505c31167-istio-data\") pod \"e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4\" (UID: \"e671c902-62e8-48fc-bf30-6a2505c31167\") " pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 20:56:50.749775 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.748395    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e671c902-62e8-48fc-bf30-6a2505c31167-istiod-ca-cert\") pod \"e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4\" (UID: \"e671c902-62e8-48fc-bf30-6a2505c31167\") " pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 20:56:50.749775 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.748641    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e671c902-62e8-48fc-bf30-6a2505c31167-credential-socket\") pod \"e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4\" (UID: \"e671c902-62e8-48fc-bf30-6a2505c31167\") " pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 20:56:50.753708 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.753680    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e671c902-62e8-48fc-bf30-6a2505c31167-istio-podinfo\") pod \"e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4\" (UID: \"e671c902-62e8-48fc-bf30-6a2505c31167\") " pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 20:56:50.755047 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.755020    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e671c902-62e8-48fc-bf30-6a2505c31167-istio-envoy\") pod \"e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4\" (UID: \"e671c902-62e8-48fc-bf30-6a2505c31167\") " pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 20:56:50.808142 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.808111    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e671c902-62e8-48fc-bf30-6a2505c31167-istio-token\") pod \"e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4\" (UID: \"e671c902-62e8-48fc-bf30-6a2505c31167\") " pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 20:56:50.808596 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.808571    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z6tw\" (UniqueName: \"kubernetes.io/projected/e671c902-62e8-48fc-bf30-6a2505c31167-kube-api-access-5z6tw\") pod \"e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4\" (UID: \"e671c902-62e8-48fc-bf30-6a2505c31167\") " pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 20:56:50.846084 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:50.846051    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 20:56:51.046940 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:51.046840    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:51.282502 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:56:51.282465    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode671c902_62e8_48fc_bf30_6a2505c31167.slice/crio-122aaf70397c99a78bf308aa9a6b8cc5c1150d31ba57ec1e445cfedadb808984 WatchSource:0}: Error finding container 122aaf70397c99a78bf308aa9a6b8cc5c1150d31ba57ec1e445cfedadb808984: Status 404 returned error can't find the container with id 122aaf70397c99a78bf308aa9a6b8cc5c1150d31ba57ec1e445cfedadb808984
Oct 30 20:56:51.283793 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:51.283766    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"]
Oct 30 20:56:51.285563 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:51.285522    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:56:51.285881 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:51.285604    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:56:51.285881 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:51.285649    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:56:51.459407 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:51.459368    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4" event={"ID":"e671c902-62e8-48fc-bf30-6a2505c31167","Type":"ContainerStarted","Data":"7154b1841877e313e01bf6c52cd7ed097f0b93e299dc64d12725273d337fdff7"}
Oct 30 20:56:51.459598 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:51.459432    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4" event={"ID":"e671c902-62e8-48fc-bf30-6a2505c31167","Type":"ContainerStarted","Data":"122aaf70397c99a78bf308aa9a6b8cc5c1150d31ba57ec1e445cfedadb808984"}
Oct 30 20:56:51.607725 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:51.607653    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4" podStartSLOduration=1.607634308 podStartE2EDuration="1.607634308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:51.603520483 +0000 UTC m=+1823.520542054" watchObservedRunningTime="2026-10-30 20:56:51.607634308 +0000 UTC m=+1823.524655846"
Oct 30 20:56:51.846269 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:51.846236    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 20:56:52.007313 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:52.007233    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:52.246489 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:52.246397    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:52.258698 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:52.258583    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm"]
Oct 30 20:56:52.259469 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:52.258966    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm" podUID="b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26" containerName="maas-api" containerID="cri-o://03a0da1ab6c7d26705bd5d8e161c7542afebd31108e09c458b1aae1dae9a08a0" gracePeriod=30
Oct 30 20:56:52.418997 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:52.418960    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:52.585866 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:52.584080    5071 generic.go:356] "Generic (PLEG): container finished" podID="b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26" containerID="03a0da1ab6c7d26705bd5d8e161c7542afebd31108e09c458b1aae1dae9a08a0" exitCode=0
Oct 30 20:56:52.585866 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:52.584802    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm" event={"ID":"b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26","Type":"ContainerDied","Data":"03a0da1ab6c7d26705bd5d8e161c7542afebd31108e09c458b1aae1dae9a08a0"}
Oct 30 20:56:52.735990 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:52.735385    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm"
Oct 30 20:56:52.852347 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:52.852274    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 20:56:52.866812 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:52.866780    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26-metrics-certs\") pod \"b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26\" (UID: \"b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26\") "
Oct 30 20:56:52.866992 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:52.866917    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26-kube-api-access-5st7b\" (UniqueName: \"kubernetes.io/projected/b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26-kube-api-access-5st7b\") pod \"b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26\" (UID: \"b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26\") "
Oct 30 20:56:52.867150 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:52.867134    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26-openshift-service-ca\") pod \"b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26\" (UID: \"b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26\") "
Oct 30 20:56:52.867227 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:52.867213    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26-maas-api-tls\") pod \"b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26\" (UID: \"b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26\") "
Oct 30 20:56:52.867559 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:52.867532    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26-openshift-service-ca" pod "b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26" (UID: "b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:56:52.867768 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:52.867750    5071 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26-openshift-service-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:52.870193 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:52.870171    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26-kube-api-access-5st7b" pod "b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26" (UID: "b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26"). InnerVolumeSpecName "kube-api-access-5st7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:56:52.870285 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:52.870195    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26-metrics-certs" pod "b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26" (UID: "b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:56:52.871562 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:52.871532    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26-maas-api-tls" pod "b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26" (UID: "b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:56:52.974319 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:52.974115    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26-maas-api-tls\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:52.974319 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:52.974150    5071 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26-metrics-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:52.974319 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:52.974169    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5st7b\" (UniqueName: \"kubernetes.io/projected/b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26-kube-api-access-5st7b\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:53.627852 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:53.620285    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm"
Oct 30 20:56:53.627852 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:53.626095    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm" event={"ID":"b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26","Type":"ContainerDied","Data":"1825750e78ea20a69e56b61fb02aa3875dca03605ffd1fff21d5b3b3c96d9fa0"}
Oct 30 20:56:53.627852 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:53.626163    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 20:56:53.627852 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:53.626197    5071 scope.go:122] "RemoveContainer" containerID="03a0da1ab6c7d26705bd5d8e161c7542afebd31108e09c458b1aae1dae9a08a0"
Oct 30 20:56:53.641390 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:53.641140    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 20:56:53.918410 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:53.918324    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm"]
Oct 30 20:56:53.922278 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:53.922248    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b86743-57946d64f8-jkgbm"]
Oct 30 20:56:54.144424 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:54.144361    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:54.203153 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:54.203073    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26" path="/var/lib/kubelet/pods/b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26/volumes"
Oct 30 20:56:54.443616 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:54.443575    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:54.609199 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:54.609143    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:55.134566 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:55.134385    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:55.309426 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:55.309394    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:55.496208 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:55.496114    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:58.465623 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:58.465587    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:58.681126 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:58.676486    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:56:58.876419 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:58.876385    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-19176d-69f48c59bc-wgtb8"]
Oct 30 20:56:58.877682 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:58.877508    5071 cpu_manager.go:414] "containerMap: removing container" podUID="b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26" containerName="maas-api"
Oct 30 20:56:58.877682 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:58.877534    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26" containerName="maas-api"
Oct 30 20:56:58.878535 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:58.878512    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3ca1ec4-c4d4-4b8b-b2d1-3faf33a4ff26" containerName="maas-api"
Oct 30 20:56:58.884152 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:58.884129    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-19176d-69f48c59bc-wgtb8"
Oct 30 20:56:58.920463 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:58.920426    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9104e5d7-09e8-4622-b783-69506c636052-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-19176d-69f48c59bc-wgtb8\" (UID: \"9104e5d7-09e8-4622-b783-69506c636052\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-19176d-69f48c59bc-wgtb8"
Oct 30 20:56:58.920602 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:58.920567    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbvsv\" (UniqueName: \"kubernetes.io/projected/9104e5d7-09e8-4622-b783-69506c636052-kube-api-access-cbvsv\") pod \"payload-pre-processing-e2e-shared-b-w4-19176d-69f48c59bc-wgtb8\" (UID: \"9104e5d7-09e8-4622-b783-69506c636052\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-19176d-69f48c59bc-wgtb8"
Oct 30 20:56:58.932775 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:58.932741    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-19176d-69f48c59bc-wgtb8"]
Oct 30 20:56:58.991847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:58.975118    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-19176d\""
Oct 30 20:56:58.991847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:58.975286    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-19176d-dockercfg-jjjs8\""
Oct 30 20:56:59.107018 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:59.099561    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbvsv\" (UniqueName: \"kubernetes.io/projected/9104e5d7-09e8-4622-b783-69506c636052-kube-api-access-cbvsv\") pod \"payload-pre-processing-e2e-shared-b-w4-19176d-69f48c59bc-wgtb8\" (UID: \"9104e5d7-09e8-4622-b783-69506c636052\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-19176d-69f48c59bc-wgtb8"
Oct 30 20:56:59.107018 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:59.099859    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9104e5d7-09e8-4622-b783-69506c636052-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-19176d-69f48c59bc-wgtb8\" (UID: \"9104e5d7-09e8-4622-b783-69506c636052\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-19176d-69f48c59bc-wgtb8"
Oct 30 20:56:59.107018 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:59.100909    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9104e5d7-09e8-4622-b783-69506c636052-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-19176d-69f48c59bc-wgtb8\" (UID: \"9104e5d7-09e8-4622-b783-69506c636052\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-19176d-69f48c59bc-wgtb8"
Oct 30 20:56:59.155911 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:59.151297    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-19176d-ff6f9d87d-5l9qg"]
Oct 30 20:56:59.159326 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:59.158118    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-19176d-ff6f9d87d-5l9qg"
Oct 30 20:56:59.165661 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:59.165622    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-19176d-ff6f9d87d-5l9qg"]
Oct 30 20:56:59.185525 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:59.185466    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbvsv\" (UniqueName: \"kubernetes.io/projected/9104e5d7-09e8-4622-b783-69506c636052-kube-api-access-cbvsv\") pod \"payload-pre-processing-e2e-shared-b-w4-19176d-69f48c59bc-wgtb8\" (UID: \"9104e5d7-09e8-4622-b783-69506c636052\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-19176d-69f48c59bc-wgtb8"
Oct 30 20:56:59.198568 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:59.198550    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-19176d-69f48c59bc-wgtb8"
Oct 30 20:56:59.200703 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:59.200680    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6a43db1-079d-4a4e-b048-641193df5ea4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-19176d-ff6f9d87d-5l9qg\" (UID: \"b6a43db1-079d-4a4e-b048-641193df5ea4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-19176d-ff6f9d87d-5l9qg"
Oct 30 20:56:59.202475 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:59.201097    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lsr8\" (UniqueName: \"kubernetes.io/projected/b6a43db1-079d-4a4e-b048-641193df5ea4-kube-api-access-2lsr8\") pod \"payload-processing-e2e-shared-b-w4-19176d-ff6f9d87d-5l9qg\" (UID: \"b6a43db1-079d-4a4e-b048-641193df5ea4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-19176d-ff6f9d87d-5l9qg"
Oct 30 20:56:59.302633 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:59.302585    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lsr8\" (UniqueName: \"kubernetes.io/projected/b6a43db1-079d-4a4e-b048-641193df5ea4-kube-api-access-2lsr8\") pod \"payload-processing-e2e-shared-b-w4-19176d-ff6f9d87d-5l9qg\" (UID: \"b6a43db1-079d-4a4e-b048-641193df5ea4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-19176d-ff6f9d87d-5l9qg"
Oct 30 20:56:59.302849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:59.302727    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6a43db1-079d-4a4e-b048-641193df5ea4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-19176d-ff6f9d87d-5l9qg\" (UID: \"b6a43db1-079d-4a4e-b048-641193df5ea4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-19176d-ff6f9d87d-5l9qg"
Oct 30 20:56:59.303792 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:59.303759    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6a43db1-079d-4a4e-b048-641193df5ea4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-19176d-ff6f9d87d-5l9qg\" (UID: \"b6a43db1-079d-4a4e-b048-641193df5ea4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-19176d-ff6f9d87d-5l9qg"
Oct 30 20:56:59.362005 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:59.361928    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lsr8\" (UniqueName: \"kubernetes.io/projected/b6a43db1-079d-4a4e-b048-641193df5ea4-kube-api-access-2lsr8\") pod \"payload-processing-e2e-shared-b-w4-19176d-ff6f9d87d-5l9qg\" (UID: \"b6a43db1-079d-4a4e-b048-641193df5ea4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-19176d-ff6f9d87d-5l9qg"
Oct 30 20:56:59.509492 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:59.509419    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-19176d-ff6f9d87d-5l9qg"
Oct 30 20:56:59.595081 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:59.594946    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-19176d-69f48c59bc-wgtb8"]
Oct 30 20:56:59.695079 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:59.695052    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-19176d-69f48c59bc-wgtb8" event={"ID":"9104e5d7-09e8-4622-b783-69506c636052","Type":"ContainerStarted","Data":"a2b16fba6163ba76353cb884a654b0b4bf5b8c7f896143edac39e222531f44ca"}
Oct 30 20:56:59.895150 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:59.895120    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-19176d-ff6f9d87d-5l9qg"]
Oct 30 20:56:59.897111 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:56:59.897085    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a43db1_079d_4a4e_b048_641193df5ea4.slice/crio-3e27fadb79bf6a1eb712fc84bcaac46cd81541ff4e7d77b4222d9457a4dbc610 WatchSource:0}: Error finding container 3e27fadb79bf6a1eb712fc84bcaac46cd81541ff4e7d77b4222d9457a4dbc610: Status 404 returned error can't find the container with id 3e27fadb79bf6a1eb712fc84bcaac46cd81541ff4e7d77b4222d9457a4dbc610
Oct 30 20:56:59.952792 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:56:59.952766    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:00.705078 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:00.705044    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-19176d-ff6f9d87d-5l9qg" event={"ID":"b6a43db1-079d-4a4e-b048-641193df5ea4","Type":"ContainerStarted","Data":"06d27e6cef247833ec88b653239a0f341dcaa36127b972ee4c377c9c42123d02"}
Oct 30 20:57:00.705529 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:00.705103    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-19176d-ff6f9d87d-5l9qg" event={"ID":"b6a43db1-079d-4a4e-b048-641193df5ea4","Type":"ContainerStarted","Data":"3e27fadb79bf6a1eb712fc84bcaac46cd81541ff4e7d77b4222d9457a4dbc610"}
Oct 30 20:57:00.705529 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:00.705133    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-19176d-ff6f9d87d-5l9qg"
Oct 30 20:57:00.707138 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:00.707107    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-19176d-69f48c59bc-wgtb8" event={"ID":"9104e5d7-09e8-4622-b783-69506c636052","Type":"ContainerStarted","Data":"77d8bf64718549233ad972f0369d7b4b6ac5b048e6e5fd33f60180b0ed2b245e"}
Oct 30 20:57:00.707263 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:00.707157    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-19176d-69f48c59bc-wgtb8"
Oct 30 20:57:00.821639 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:00.821583    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-19176d-ff6f9d87d-5l9qg" podStartSLOduration=2.82156971 podStartE2EDuration="2.82156971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:00.819932308 +0000 UTC m=+1832.736953845" watchObservedRunningTime="2026-10-30 20:57:00.82156971 +0000 UTC m=+1832.738591262"
Oct 30 20:57:02.077845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:02.077791    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:02.280046 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:02.280013    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:02.376099 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:02.375970    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-19176d-69f48c59bc-wgtb8" podStartSLOduration=4.375948542 podStartE2EDuration="4.375948542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:00.969121761 +0000 UTC m=+1832.886143298" watchObservedRunningTime="2026-10-30 20:57:02.375948542 +0000 UTC m=+1834.292970079"
Oct 30 20:57:02.381287 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:02.381257    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"]
Oct 30 20:57:02.381546 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:02.381520    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx" podUID="f34c7327-17c9-4085-8970-27dad43ee9be" containerName="istio-proxy" containerID="cri-o://6e2c482fe042adf988fd644bae9f0fc85ab95e78f063b0536de8b7cc5d9237dc" gracePeriod=30
Oct 30 20:57:02.447053 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:02.446963    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:04.207363 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:04.207327    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"]
Oct 30 20:57:04.208539 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:04.208452    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l" podUID="10329fd9-4ea0-41ac-a93b-6a0237b5f7c1" containerName="istio-proxy" containerID="cri-o://554e99148efcd61cf3d5a180bdaf3ee851a735139dce85de7546fcf330fda033" gracePeriod=30
Oct 30 20:57:05.376841 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:05.376772    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:05.813839 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:05.813788    5071 patch_prober.go:29] interesting pod/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:57:05.814036 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:05.813891    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l" podUID="10329fd9-4ea0-41ac-a93b-6a0237b5f7c1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:57:06.044020 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:06.043987    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:57:07.675976 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.675952    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:57:07.717170 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.717126    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f34c7327-17c9-4085-8970-27dad43ee9be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34c7327-17c9-4085-8970-27dad43ee9be-credential-socket\") pod \"f34c7327-17c9-4085-8970-27dad43ee9be\" (UID: \"f34c7327-17c9-4085-8970-27dad43ee9be\") "
Oct 30 20:57:07.717322 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.717245    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f34c7327-17c9-4085-8970-27dad43ee9be-kube-api-access-bzv92\" (UniqueName: \"kubernetes.io/projected/f34c7327-17c9-4085-8970-27dad43ee9be-kube-api-access-bzv92\") pod \"f34c7327-17c9-4085-8970-27dad43ee9be\" (UID: \"f34c7327-17c9-4085-8970-27dad43ee9be\") "
Oct 30 20:57:07.717381 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.717331    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f34c7327-17c9-4085-8970-27dad43ee9be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f34c7327-17c9-4085-8970-27dad43ee9be-istio-podinfo\") pod \"f34c7327-17c9-4085-8970-27dad43ee9be\" (UID: \"f34c7327-17c9-4085-8970-27dad43ee9be\") "
Oct 30 20:57:07.717432 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.717408    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f34c7327-17c9-4085-8970-27dad43ee9be-istio-token\" (UniqueName: \"kubernetes.io/projected/f34c7327-17c9-4085-8970-27dad43ee9be-istio-token\") pod \"f34c7327-17c9-4085-8970-27dad43ee9be\" (UID: \"f34c7327-17c9-4085-8970-27dad43ee9be\") "
Oct 30 20:57:07.717474 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.717438    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f34c7327-17c9-4085-8970-27dad43ee9be-credential-socket" pod "f34c7327-17c9-4085-8970-27dad43ee9be" (UID: "f34c7327-17c9-4085-8970-27dad43ee9be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:07.717532 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.717516    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f34c7327-17c9-4085-8970-27dad43ee9be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f34c7327-17c9-4085-8970-27dad43ee9be-istiod-ca-cert\") pod \"f34c7327-17c9-4085-8970-27dad43ee9be\" (UID: \"f34c7327-17c9-4085-8970-27dad43ee9be\") "
Oct 30 20:57:07.717574 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.717565    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f34c7327-17c9-4085-8970-27dad43ee9be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f34c7327-17c9-4085-8970-27dad43ee9be-istio-data\") pod \"f34c7327-17c9-4085-8970-27dad43ee9be\" (UID: \"f34c7327-17c9-4085-8970-27dad43ee9be\") "
Oct 30 20:57:07.717635 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.717610    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f34c7327-17c9-4085-8970-27dad43ee9be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f34c7327-17c9-4085-8970-27dad43ee9be-istio-envoy\") pod \"f34c7327-17c9-4085-8970-27dad43ee9be\" (UID: \"f34c7327-17c9-4085-8970-27dad43ee9be\") "
Oct 30 20:57:07.717689 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.717666    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f34c7327-17c9-4085-8970-27dad43ee9be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f34c7327-17c9-4085-8970-27dad43ee9be-workload-certs\") pod \"f34c7327-17c9-4085-8970-27dad43ee9be\" (UID: \"f34c7327-17c9-4085-8970-27dad43ee9be\") "
Oct 30 20:57:07.717774 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.717749    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f34c7327-17c9-4085-8970-27dad43ee9be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34c7327-17c9-4085-8970-27dad43ee9be-workload-socket\") pod \"f34c7327-17c9-4085-8970-27dad43ee9be\" (UID: \"f34c7327-17c9-4085-8970-27dad43ee9be\") "
Oct 30 20:57:07.717965 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.717942    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f34c7327-17c9-4085-8970-27dad43ee9be-istiod-ca-cert" pod "f34c7327-17c9-4085-8970-27dad43ee9be" (UID: "f34c7327-17c9-4085-8970-27dad43ee9be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:07.718295 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.718218    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f34c7327-17c9-4085-8970-27dad43ee9be-workload-certs" pod "f34c7327-17c9-4085-8970-27dad43ee9be" (UID: "f34c7327-17c9-4085-8970-27dad43ee9be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:07.718423 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.718323    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f34c7327-17c9-4085-8970-27dad43ee9be-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:07.718423 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.718349    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f34c7327-17c9-4085-8970-27dad43ee9be-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:07.718423 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.718359    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34c7327-17c9-4085-8970-27dad43ee9be-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:07.718423 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.718319    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f34c7327-17c9-4085-8970-27dad43ee9be-workload-socket" pod "f34c7327-17c9-4085-8970-27dad43ee9be" (UID: "f34c7327-17c9-4085-8970-27dad43ee9be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:07.719094 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.719052    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f34c7327-17c9-4085-8970-27dad43ee9be-istio-data" pod "f34c7327-17c9-4085-8970-27dad43ee9be" (UID: "f34c7327-17c9-4085-8970-27dad43ee9be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:07.719862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.719804    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f34c7327-17c9-4085-8970-27dad43ee9be-istio-token" pod "f34c7327-17c9-4085-8970-27dad43ee9be" (UID: "f34c7327-17c9-4085-8970-27dad43ee9be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:07.719956 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.719879    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f34c7327-17c9-4085-8970-27dad43ee9be-istio-podinfo" pod "f34c7327-17c9-4085-8970-27dad43ee9be" (UID: "f34c7327-17c9-4085-8970-27dad43ee9be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:57:07.720056 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.720041    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f34c7327-17c9-4085-8970-27dad43ee9be-kube-api-access-bzv92" pod "f34c7327-17c9-4085-8970-27dad43ee9be" (UID: "f34c7327-17c9-4085-8970-27dad43ee9be"). InnerVolumeSpecName "kube-api-access-bzv92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:07.721016 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.720995    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f34c7327-17c9-4085-8970-27dad43ee9be-istio-envoy" pod "f34c7327-17c9-4085-8970-27dad43ee9be" (UID: "f34c7327-17c9-4085-8970-27dad43ee9be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:07.766774 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.766747    5071 generic.go:356] "Generic (PLEG): container finished" podID="f34c7327-17c9-4085-8970-27dad43ee9be" containerID="6e2c482fe042adf988fd644bae9f0fc85ab95e78f063b0536de8b7cc5d9237dc" exitCode=0
Oct 30 20:57:07.766897 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.766798    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"
Oct 30 20:57:07.766897 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.766803    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx" event={"ID":"f34c7327-17c9-4085-8970-27dad43ee9be","Type":"ContainerDied","Data":"6e2c482fe042adf988fd644bae9f0fc85ab95e78f063b0536de8b7cc5d9237dc"}
Oct 30 20:57:07.766987 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.766925    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx" event={"ID":"f34c7327-17c9-4085-8970-27dad43ee9be","Type":"ContainerDied","Data":"16d7f48ebd3d1fe2b2dcb0fe8a7f1fe67eba0efe495b83cf21dec5e2fdf70eb7"}
Oct 30 20:57:07.766987 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.766956    5071 scope.go:122] "RemoveContainer" containerID="6e2c482fe042adf988fd644bae9f0fc85ab95e78f063b0536de8b7cc5d9237dc"
Oct 30 20:57:07.777867 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.777852    5071 scope.go:122] "RemoveContainer" containerID="73eedcc882576a85936c0f80f504e05da537151d645a1ee4893772a31386424b"
Oct 30 20:57:07.788318 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.788298    5071 scope.go:122] "RemoveContainer" containerID="6e2c482fe042adf988fd644bae9f0fc85ab95e78f063b0536de8b7cc5d9237dc"
Oct 30 20:57:07.788568 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:57:07.788550    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e2c482fe042adf988fd644bae9f0fc85ab95e78f063b0536de8b7cc5d9237dc\": container with ID starting with 6e2c482fe042adf988fd644bae9f0fc85ab95e78f063b0536de8b7cc5d9237dc not found: ID does not exist" containerID="6e2c482fe042adf988fd644bae9f0fc85ab95e78f063b0536de8b7cc5d9237dc"
Oct 30 20:57:07.788629 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.788576    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e2c482fe042adf988fd644bae9f0fc85ab95e78f063b0536de8b7cc5d9237dc"} err="failed to get container status \"6e2c482fe042adf988fd644bae9f0fc85ab95e78f063b0536de8b7cc5d9237dc\": rpc error: code = NotFound desc = could not find container \"6e2c482fe042adf988fd644bae9f0fc85ab95e78f063b0536de8b7cc5d9237dc\": container with ID starting with 6e2c482fe042adf988fd644bae9f0fc85ab95e78f063b0536de8b7cc5d9237dc not found: ID does not exist"
Oct 30 20:57:07.788629 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.788593    5071 scope.go:122] "RemoveContainer" containerID="73eedcc882576a85936c0f80f504e05da537151d645a1ee4893772a31386424b"
Oct 30 20:57:07.788809 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:57:07.788790    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73eedcc882576a85936c0f80f504e05da537151d645a1ee4893772a31386424b\": container with ID starting with 73eedcc882576a85936c0f80f504e05da537151d645a1ee4893772a31386424b not found: ID does not exist" containerID="73eedcc882576a85936c0f80f504e05da537151d645a1ee4893772a31386424b"
Oct 30 20:57:07.788881 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.788815    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73eedcc882576a85936c0f80f504e05da537151d645a1ee4893772a31386424b"} err="failed to get container status \"73eedcc882576a85936c0f80f504e05da537151d645a1ee4893772a31386424b\": rpc error: code = NotFound desc = could not find container \"73eedcc882576a85936c0f80f504e05da537151d645a1ee4893772a31386424b\": container with ID starting with 73eedcc882576a85936c0f80f504e05da537151d645a1ee4893772a31386424b not found: ID does not exist"
Oct 30 20:57:07.819795 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.819750    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f34c7327-17c9-4085-8970-27dad43ee9be-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:07.819795 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.819774    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f34c7327-17c9-4085-8970-27dad43ee9be-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:07.819795 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.819783    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34c7327-17c9-4085-8970-27dad43ee9be-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:07.819795 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.819796    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzv92\" (UniqueName: \"kubernetes.io/projected/f34c7327-17c9-4085-8970-27dad43ee9be-kube-api-access-bzv92\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:07.820078 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.819808    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f34c7327-17c9-4085-8970-27dad43ee9be-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:07.820078 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.819837    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f34c7327-17c9-4085-8970-27dad43ee9be-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:07.965974 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.965897    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"]
Oct 30 20:57:07.969281 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:07.969259    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b86743-openshift-default-8689f7d54c-ds9wx"]
Oct 30 20:57:08.203783 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:08.203751    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f34c7327-17c9-4085-8970-27dad43ee9be" path="/var/lib/kubelet/pods/f34c7327-17c9-4085-8970-27dad43ee9be/volumes"
Oct 30 20:57:09.514377 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:09.514349    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"
Oct 30 20:57:09.643278 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:09.643246    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-istio-data\") pod \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\" (UID: \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\") "
Oct 30 20:57:09.643486 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:09.643343    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-istio-token\" (UniqueName: \"kubernetes.io/projected/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-istio-token\") pod \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\" (UID: \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\") "
Oct 30 20:57:09.643486 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:09.643384    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-credential-socket\") pod \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\" (UID: \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\") "
Oct 30 20:57:09.643486 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:09.643433    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-workload-socket\") pod \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\" (UID: \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\") "
Oct 30 20:57:09.643486 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:09.643459    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-istio-podinfo\") pod \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\" (UID: \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\") "
Oct 30 20:57:09.643710 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:09.643507    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-kube-api-access-5wr7j\" (UniqueName: \"kubernetes.io/projected/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-kube-api-access-5wr7j\") pod \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\" (UID: \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\") "
Oct 30 20:57:09.643710 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:09.643540    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-workload-certs\") pod \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\" (UID: \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\") "
Oct 30 20:57:09.643710 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:09.643573    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-istio-envoy\") pod \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\" (UID: \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\") "
Oct 30 20:57:09.643710 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:09.643589    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-istiod-ca-cert\") pod \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\" (UID: \"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1\") "
Oct 30 20:57:09.644567 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:09.644490    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-workload-socket" pod "10329fd9-4ea0-41ac-a93b-6a0237b5f7c1" (UID: "10329fd9-4ea0-41ac-a93b-6a0237b5f7c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:09.644672 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:09.644600    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-credential-socket" pod "10329fd9-4ea0-41ac-a93b-6a0237b5f7c1" (UID: "10329fd9-4ea0-41ac-a93b-6a0237b5f7c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:09.644898 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:09.644734    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-workload-certs" pod "10329fd9-4ea0-41ac-a93b-6a0237b5f7c1" (UID: "10329fd9-4ea0-41ac-a93b-6a0237b5f7c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:09.645009 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:09.644953    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-istio-data" pod "10329fd9-4ea0-41ac-a93b-6a0237b5f7c1" (UID: "10329fd9-4ea0-41ac-a93b-6a0237b5f7c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:09.645009 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:09.644981    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-istiod-ca-cert" pod "10329fd9-4ea0-41ac-a93b-6a0237b5f7c1" (UID: "10329fd9-4ea0-41ac-a93b-6a0237b5f7c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:09.647014 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:09.646988    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-kube-api-access-5wr7j" pod "10329fd9-4ea0-41ac-a93b-6a0237b5f7c1" (UID: "10329fd9-4ea0-41ac-a93b-6a0237b5f7c1"). InnerVolumeSpecName "kube-api-access-5wr7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:09.647369 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:09.647343    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-istio-podinfo" pod "10329fd9-4ea0-41ac-a93b-6a0237b5f7c1" (UID: "10329fd9-4ea0-41ac-a93b-6a0237b5f7c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:57:09.647676 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:09.647644    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-istio-envoy" pod "10329fd9-4ea0-41ac-a93b-6a0237b5f7c1" (UID: "10329fd9-4ea0-41ac-a93b-6a0237b5f7c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:09.648351 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:09.648325    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-istio-token" pod "10329fd9-4ea0-41ac-a93b-6a0237b5f7c1" (UID: "10329fd9-4ea0-41ac-a93b-6a0237b5f7c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:09.749110 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:09.749023    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:09.749110 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:09.749060    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:09.749110 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:09.749078    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:09.749110 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:09.749093    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:09.749110 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:09.749106    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:09.749587 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:09.749120    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wr7j\" (UniqueName: \"kubernetes.io/projected/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-kube-api-access-5wr7j\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:09.749587 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:09.749134    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:09.749587 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:09.749147    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:09.749587 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:09.749161    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:09.787460 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:09.787427    5071 generic.go:356] "Generic (PLEG): container finished" podID="10329fd9-4ea0-41ac-a93b-6a0237b5f7c1" containerID="554e99148efcd61cf3d5a180bdaf3ee851a735139dce85de7546fcf330fda033" exitCode=0
Oct 30 20:57:09.787633 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:09.787471    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l" event={"ID":"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1","Type":"ContainerDied","Data":"554e99148efcd61cf3d5a180bdaf3ee851a735139dce85de7546fcf330fda033"}
Oct 30 20:57:09.787633 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:09.787493    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"
Oct 30 20:57:09.787633 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:09.787570    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l" event={"ID":"10329fd9-4ea0-41ac-a93b-6a0237b5f7c1","Type":"ContainerDied","Data":"0285fee41de1147c18f87a0f78e07ba001483102763ad44961084303fe2388f0"}
Oct 30 20:57:09.787633 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:09.787597    5071 scope.go:122] "RemoveContainer" containerID="554e99148efcd61cf3d5a180bdaf3ee851a735139dce85de7546fcf330fda033"
Oct 30 20:57:09.799773 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:09.799755    5071 scope.go:122] "RemoveContainer" containerID="554e99148efcd61cf3d5a180bdaf3ee851a735139dce85de7546fcf330fda033"
Oct 30 20:57:09.800089 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:57:09.800068    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"554e99148efcd61cf3d5a180bdaf3ee851a735139dce85de7546fcf330fda033\": container with ID starting with 554e99148efcd61cf3d5a180bdaf3ee851a735139dce85de7546fcf330fda033 not found: ID does not exist" containerID="554e99148efcd61cf3d5a180bdaf3ee851a735139dce85de7546fcf330fda033"
Oct 30 20:57:09.800151 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:09.800097    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"554e99148efcd61cf3d5a180bdaf3ee851a735139dce85de7546fcf330fda033"} err="failed to get container status \"554e99148efcd61cf3d5a180bdaf3ee851a735139dce85de7546fcf330fda033\": rpc error: code = NotFound desc = could not find container \"554e99148efcd61cf3d5a180bdaf3ee851a735139dce85de7546fcf330fda033\": container with ID starting with 554e99148efcd61cf3d5a180bdaf3ee851a735139dce85de7546fcf330fda033 not found: ID does not exist"
Oct 30 20:57:09.959041 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:09.959000    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"]
Oct 30 20:57:09.966169 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:09.966142    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-6pp7l"]
Oct 30 20:57:10.203014 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:10.202986    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10329fd9-4ea0-41ac-a93b-6a0237b5f7c1" path="/var/lib/kubelet/pods/10329fd9-4ea0-41ac-a93b-6a0237b5f7c1/volumes"
Oct 30 20:57:10.779484 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:10.779408    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:11.717437 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:11.717407    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-19176d-69f48c59bc-wgtb8"
Oct 30 20:57:11.717614 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:11.717482    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-19176d-ff6f9d87d-5l9qg"
Oct 30 20:57:12.053222 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:12.053182    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-822bcb11-6rk2l"]
Oct 30 20:57:12.059079 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:12.059046    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-822bcb11-6rk2l"]
Oct 30 20:57:12.202878 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:12.202837    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5" path="/var/lib/kubelet/pods/acc60fa5-cc15-41bd-b1a0-eee4ebf4a3d5/volumes"
Oct 30 20:57:14.770908 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:14.770872    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-aaf669-5994948fb8-5tw77"]
Oct 30 20:57:14.887324 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:14.887290    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aaf669-656d468487-p5q8l"]
Oct 30 20:57:15.154080 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:15.154034    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:15.307473 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:15.307440    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:16.069291 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:16.069254    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:16.729900 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:57:16.729864    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:57:17.707853 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:17.707794    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:18.004287 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:18.004201    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:19.771198 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:19.771155    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-aaf669-5994948fb8-5tw77" podUID="cce1c70d-cedf-494d-9abf-202a30a44263" containerName="payload-processing" containerID="cri-o://e93391e93283ae81ca8ff2785493335e08544d52f8cf775a6e7af1369c548f68" gracePeriod=25
Oct 30 20:57:19.888356 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:19.888298    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aaf669-656d468487-p5q8l" podUID="147e612e-f84b-4696-ade0-bb3ba142dd4a" containerName="payload-pre-processing" containerID="cri-o://611c7f32d3fd30e4e97701beadb28b180545014839ae53f3e525f2b94d2c94bc" gracePeriod=25
Oct 30 20:57:19.969647 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:19.969616    5071 generic.go:356] "Generic (PLEG): container finished" podID="cce1c70d-cedf-494d-9abf-202a30a44263" containerID="e93391e93283ae81ca8ff2785493335e08544d52f8cf775a6e7af1369c548f68" exitCode=0
Oct 30 20:57:19.969851 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:19.969729    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-aaf669-5994948fb8-5tw77" event={"ID":"cce1c70d-cedf-494d-9abf-202a30a44263","Type":"ContainerDied","Data":"e93391e93283ae81ca8ff2785493335e08544d52f8cf775a6e7af1369c548f68"}
Oct 30 20:57:20.078484 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:20.078047    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-aaf669-5994948fb8-5tw77"
Oct 30 20:57:20.191606 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:20.191576    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aaf669-656d468487-p5q8l"
Oct 30 20:57:20.248776 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:20.248738    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cce1c70d-cedf-494d-9abf-202a30a44263-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cce1c70d-cedf-494d-9abf-202a30a44263-plugins-config-volume\") pod \"cce1c70d-cedf-494d-9abf-202a30a44263\" (UID: \"cce1c70d-cedf-494d-9abf-202a30a44263\") "
Oct 30 20:57:20.248963 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:20.248781    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cce1c70d-cedf-494d-9abf-202a30a44263-kube-api-access-ltklg\" (UniqueName: \"kubernetes.io/projected/cce1c70d-cedf-494d-9abf-202a30a44263-kube-api-access-ltklg\") pod \"cce1c70d-cedf-494d-9abf-202a30a44263\" (UID: \"cce1c70d-cedf-494d-9abf-202a30a44263\") "
Oct 30 20:57:20.249195 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:20.249168    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cce1c70d-cedf-494d-9abf-202a30a44263-plugins-config-volume" pod "cce1c70d-cedf-494d-9abf-202a30a44263" (UID: "cce1c70d-cedf-494d-9abf-202a30a44263"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:20.249395 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:20.249377    5071 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cce1c70d-cedf-494d-9abf-202a30a44263-plugins-config-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:20.251247 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:20.251221    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce1c70d-cedf-494d-9abf-202a30a44263-kube-api-access-ltklg" pod "cce1c70d-cedf-494d-9abf-202a30a44263" (UID: "cce1c70d-cedf-494d-9abf-202a30a44263"). InnerVolumeSpecName "kube-api-access-ltklg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:20.350845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:20.350742    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/147e612e-f84b-4696-ade0-bb3ba142dd4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147e612e-f84b-4696-ade0-bb3ba142dd4a-plugins-config-volume\") pod \"147e612e-f84b-4696-ade0-bb3ba142dd4a\" (UID: \"147e612e-f84b-4696-ade0-bb3ba142dd4a\") "
Oct 30 20:57:20.351004 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:20.350874    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/147e612e-f84b-4696-ade0-bb3ba142dd4a-kube-api-access-thqkd\" (UniqueName: \"kubernetes.io/projected/147e612e-f84b-4696-ade0-bb3ba142dd4a-kube-api-access-thqkd\") pod \"147e612e-f84b-4696-ade0-bb3ba142dd4a\" (UID: \"147e612e-f84b-4696-ade0-bb3ba142dd4a\") "
Oct 30 20:57:20.351143 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:20.351120    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/147e612e-f84b-4696-ade0-bb3ba142dd4a-plugins-config-volume" pod "147e612e-f84b-4696-ade0-bb3ba142dd4a" (UID: "147e612e-f84b-4696-ade0-bb3ba142dd4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:20.351320 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:20.351303    5071 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147e612e-f84b-4696-ade0-bb3ba142dd4a-plugins-config-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:20.351354 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:20.351324    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltklg\" (UniqueName: \"kubernetes.io/projected/cce1c70d-cedf-494d-9abf-202a30a44263-kube-api-access-ltklg\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:20.353082 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:20.353057    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147e612e-f84b-4696-ade0-bb3ba142dd4a-kube-api-access-thqkd" pod "147e612e-f84b-4696-ade0-bb3ba142dd4a" (UID: "147e612e-f84b-4696-ade0-bb3ba142dd4a"). InnerVolumeSpecName "kube-api-access-thqkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:20.452427 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:20.452386    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thqkd\" (UniqueName: \"kubernetes.io/projected/147e612e-f84b-4696-ade0-bb3ba142dd4a-kube-api-access-thqkd\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:20.978137 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:20.978100    5071 generic.go:356] "Generic (PLEG): container finished" podID="147e612e-f84b-4696-ade0-bb3ba142dd4a" containerID="611c7f32d3fd30e4e97701beadb28b180545014839ae53f3e525f2b94d2c94bc" exitCode=0
Oct 30 20:57:20.978609 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:20.978146    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aaf669-656d468487-p5q8l"
Oct 30 20:57:20.978609 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:20.978218    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aaf669-656d468487-p5q8l" event={"ID":"147e612e-f84b-4696-ade0-bb3ba142dd4a","Type":"ContainerDied","Data":"611c7f32d3fd30e4e97701beadb28b180545014839ae53f3e525f2b94d2c94bc"}
Oct 30 20:57:20.978609 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:20.978316    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aaf669-656d468487-p5q8l" event={"ID":"147e612e-f84b-4696-ade0-bb3ba142dd4a","Type":"ContainerDied","Data":"1449c76aa6ab084d669f73369cf099c3f2dac5196e9bf66bb18de7273b6eb2fd"}
Oct 30 20:57:20.978609 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:20.978350    5071 scope.go:122] "RemoveContainer" containerID="611c7f32d3fd30e4e97701beadb28b180545014839ae53f3e525f2b94d2c94bc"
Oct 30 20:57:20.980154 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:20.980135    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-aaf669-5994948fb8-5tw77"
Oct 30 20:57:20.980284 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:20.980194    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-aaf669-5994948fb8-5tw77" event={"ID":"cce1c70d-cedf-494d-9abf-202a30a44263","Type":"ContainerDied","Data":"c0555e369fe47e089e7ecd6483f3fdd81a200c232bff0f785973b56ef44826fb"}
Oct 30 20:57:20.990349 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:20.990329    5071 scope.go:122] "RemoveContainer" containerID="611c7f32d3fd30e4e97701beadb28b180545014839ae53f3e525f2b94d2c94bc"
Oct 30 20:57:20.990648 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:57:20.990615    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"611c7f32d3fd30e4e97701beadb28b180545014839ae53f3e525f2b94d2c94bc\": container with ID starting with 611c7f32d3fd30e4e97701beadb28b180545014839ae53f3e525f2b94d2c94bc not found: ID does not exist" containerID="611c7f32d3fd30e4e97701beadb28b180545014839ae53f3e525f2b94d2c94bc"
Oct 30 20:57:20.990737 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:20.990655    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"611c7f32d3fd30e4e97701beadb28b180545014839ae53f3e525f2b94d2c94bc"} err="failed to get container status \"611c7f32d3fd30e4e97701beadb28b180545014839ae53f3e525f2b94d2c94bc\": rpc error: code = NotFound desc = could not find container \"611c7f32d3fd30e4e97701beadb28b180545014839ae53f3e525f2b94d2c94bc\": container with ID starting with 611c7f32d3fd30e4e97701beadb28b180545014839ae53f3e525f2b94d2c94bc not found: ID does not exist"
Oct 30 20:57:20.990737 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:20.990680    5071 scope.go:122] "RemoveContainer" containerID="e93391e93283ae81ca8ff2785493335e08544d52f8cf775a6e7af1369c548f68"
Oct 30 20:57:21.152193 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:21.152112    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aaf669-656d468487-p5q8l"]
Oct 30 20:57:21.157404 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:21.157377    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aaf669-656d468487-p5q8l"]
Oct 30 20:57:21.219361 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:21.217993    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-aaf669-656d468487-p5q8l" podUID="147e612e-f84b-4696-ade0-bb3ba142dd4a" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.160:50052\" within 1s: context deadline exceeded"
Oct 30 20:57:21.219361 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:57:21.218388    5071 logging.go:55] [core] [Channel #4980 SubChannel #4981]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.160:50052", ServerName: "10.132.0.160:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.160:50052: operation was canceled"
Oct 30 20:57:21.223039 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:21.221066    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:21.274307 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:21.274276    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-aaf669-5994948fb8-5tw77"]
Oct 30 20:57:21.280287 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:21.280257    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-aaf669-5994948fb8-5tw77"]
Oct 30 20:57:22.202558 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:22.202522    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="147e612e-f84b-4696-ade0-bb3ba142dd4a" path="/var/lib/kubelet/pods/147e612e-f84b-4696-ade0-bb3ba142dd4a/volumes"
Oct 30 20:57:22.203016 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:22.202984    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cce1c70d-cedf-494d-9abf-202a30a44263" path="/var/lib/kubelet/pods/cce1c70d-cedf-494d-9abf-202a30a44263/volumes"
Oct 30 20:57:25.370021 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:25.369989    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws"]
Oct 30 20:57:25.370608 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:25.370201    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws" podUID="6bac9ec7-daeb-43cd-afb4-c8aca96f77a2" containerName="maas-api" containerID="cri-o://b490c6207f4c2258bdb7e8f737e745cd67ff200ec14f77324426032d8b755c1c" gracePeriod=30
Oct 30 20:57:25.492548 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:25.492514    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:25.732048 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:25.732026    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws"
Oct 30 20:57:25.821847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:25.821675    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bac9ec7-daeb-43cd-afb4-c8aca96f77a2-kube-api-access-qc7w8\" (UniqueName: \"kubernetes.io/projected/6bac9ec7-daeb-43cd-afb4-c8aca96f77a2-kube-api-access-qc7w8\") pod \"6bac9ec7-daeb-43cd-afb4-c8aca96f77a2\" (UID: \"6bac9ec7-daeb-43cd-afb4-c8aca96f77a2\") "
Oct 30 20:57:25.821847 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:25.821798    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bac9ec7-daeb-43cd-afb4-c8aca96f77a2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bac9ec7-daeb-43cd-afb4-c8aca96f77a2-maas-api-tls\") pod \"6bac9ec7-daeb-43cd-afb4-c8aca96f77a2\" (UID: \"6bac9ec7-daeb-43cd-afb4-c8aca96f77a2\") "
Oct 30 20:57:25.822107 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:25.821924    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bac9ec7-daeb-43cd-afb4-c8aca96f77a2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bac9ec7-daeb-43cd-afb4-c8aca96f77a2-openshift-service-ca\") pod \"6bac9ec7-daeb-43cd-afb4-c8aca96f77a2\" (UID: \"6bac9ec7-daeb-43cd-afb4-c8aca96f77a2\") "
Oct 30 20:57:25.822107 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:25.822009    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bac9ec7-daeb-43cd-afb4-c8aca96f77a2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bac9ec7-daeb-43cd-afb4-c8aca96f77a2-metrics-certs\") pod \"6bac9ec7-daeb-43cd-afb4-c8aca96f77a2\" (UID: \"6bac9ec7-daeb-43cd-afb4-c8aca96f77a2\") "
Oct 30 20:57:25.825583 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:25.824154    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bac9ec7-daeb-43cd-afb4-c8aca96f77a2-openshift-service-ca" pod "6bac9ec7-daeb-43cd-afb4-c8aca96f77a2" (UID: "6bac9ec7-daeb-43cd-afb4-c8aca96f77a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:25.843836 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:25.842388    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bac9ec7-daeb-43cd-afb4-c8aca96f77a2-maas-api-tls" pod "6bac9ec7-daeb-43cd-afb4-c8aca96f77a2" (UID: "6bac9ec7-daeb-43cd-afb4-c8aca96f77a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:57:25.848780 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:25.848042    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bac9ec7-daeb-43cd-afb4-c8aca96f77a2-metrics-certs" pod "6bac9ec7-daeb-43cd-afb4-c8aca96f77a2" (UID: "6bac9ec7-daeb-43cd-afb4-c8aca96f77a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:57:25.848780 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:25.848219    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bac9ec7-daeb-43cd-afb4-c8aca96f77a2-kube-api-access-qc7w8" pod "6bac9ec7-daeb-43cd-afb4-c8aca96f77a2" (UID: "6bac9ec7-daeb-43cd-afb4-c8aca96f77a2"). InnerVolumeSpecName "kube-api-access-qc7w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:25.923587 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:25.923556    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc7w8\" (UniqueName: \"kubernetes.io/projected/6bac9ec7-daeb-43cd-afb4-c8aca96f77a2-kube-api-access-qc7w8\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:25.923587 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:25.923584    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bac9ec7-daeb-43cd-afb4-c8aca96f77a2-maas-api-tls\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:25.923848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:25.923596    5071 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bac9ec7-daeb-43cd-afb4-c8aca96f77a2-openshift-service-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:25.923848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:25.923608    5071 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bac9ec7-daeb-43cd-afb4-c8aca96f77a2-metrics-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:26.052701 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:26.052668    5071 generic.go:356] "Generic (PLEG): container finished" podID="6bac9ec7-daeb-43cd-afb4-c8aca96f77a2" containerID="b490c6207f4c2258bdb7e8f737e745cd67ff200ec14f77324426032d8b755c1c" exitCode=0
Oct 30 20:57:26.052948 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:26.052723    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws"
Oct 30 20:57:26.052948 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:26.052757    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws" event={"ID":"6bac9ec7-daeb-43cd-afb4-c8aca96f77a2","Type":"ContainerDied","Data":"b490c6207f4c2258bdb7e8f737e745cd67ff200ec14f77324426032d8b755c1c"}
Oct 30 20:57:26.052948 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:26.052866    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws" event={"ID":"6bac9ec7-daeb-43cd-afb4-c8aca96f77a2","Type":"ContainerDied","Data":"96c78513a64286df4f6d8d77def63b73ca9a4a72e971f5e37d49b85ae59bf822"}
Oct 30 20:57:26.052948 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:26.052900    5071 scope.go:122] "RemoveContainer" containerID="b490c6207f4c2258bdb7e8f737e745cd67ff200ec14f77324426032d8b755c1c"
Oct 30 20:57:26.067371 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:26.067351    5071 scope.go:122] "RemoveContainer" containerID="b490c6207f4c2258bdb7e8f737e745cd67ff200ec14f77324426032d8b755c1c"
Oct 30 20:57:26.067664 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:57:26.067646    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b490c6207f4c2258bdb7e8f737e745cd67ff200ec14f77324426032d8b755c1c\": container with ID starting with b490c6207f4c2258bdb7e8f737e745cd67ff200ec14f77324426032d8b755c1c not found: ID does not exist" containerID="b490c6207f4c2258bdb7e8f737e745cd67ff200ec14f77324426032d8b755c1c"
Oct 30 20:57:26.067728 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:26.067676    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b490c6207f4c2258bdb7e8f737e745cd67ff200ec14f77324426032d8b755c1c"} err="failed to get container status \"b490c6207f4c2258bdb7e8f737e745cd67ff200ec14f77324426032d8b755c1c\": rpc error: code = NotFound desc = could not find container \"b490c6207f4c2258bdb7e8f737e745cd67ff200ec14f77324426032d8b755c1c\": container with ID starting with b490c6207f4c2258bdb7e8f737e745cd67ff200ec14f77324426032d8b755c1c not found: ID does not exist"
Oct 30 20:57:26.220750 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:26.220715    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws"]
Oct 30 20:57:26.225928 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:26.225903    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0702ff-7dfc6c9858-srrws"]
Oct 30 20:57:27.017661 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.017623    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x"]
Oct 30 20:57:27.018590 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.018572    5071 cpu_manager.go:414] "containerMap: removing container" podUID="10329fd9-4ea0-41ac-a93b-6a0237b5f7c1" containerName="istio-proxy"
Oct 30 20:57:27.018649 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.018593    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10329fd9-4ea0-41ac-a93b-6a0237b5f7c1" containerName="istio-proxy"
Oct 30 20:57:27.018649 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.018622    5071 cpu_manager.go:414] "containerMap: removing container" podUID="147e612e-f84b-4696-ade0-bb3ba142dd4a" containerName="payload-pre-processing"
Oct 30 20:57:27.018649 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.018630    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="147e612e-f84b-4696-ade0-bb3ba142dd4a" containerName="payload-pre-processing"
Oct 30 20:57:27.018745 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.018651    5071 cpu_manager.go:414] "containerMap: removing container" podUID="f34c7327-17c9-4085-8970-27dad43ee9be" containerName="istio-proxy"
Oct 30 20:57:27.018745 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.018658    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f34c7327-17c9-4085-8970-27dad43ee9be" containerName="istio-proxy"
Oct 30 20:57:27.018745 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.018681    5071 cpu_manager.go:414] "containerMap: removing container" podUID="cce1c70d-cedf-494d-9abf-202a30a44263" containerName="payload-processing"
Oct 30 20:57:27.018745 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.018687    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cce1c70d-cedf-494d-9abf-202a30a44263" containerName="payload-processing"
Oct 30 20:57:27.018745 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.018734    5071 cpu_manager.go:414] "containerMap: removing container" podUID="6bac9ec7-daeb-43cd-afb4-c8aca96f77a2" containerName="maas-api"
Oct 30 20:57:27.018745 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.018742    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bac9ec7-daeb-43cd-afb4-c8aca96f77a2" containerName="maas-api"
Oct 30 20:57:27.018950 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.018895    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="147e612e-f84b-4696-ade0-bb3ba142dd4a" containerName="payload-pre-processing"
Oct 30 20:57:27.018950 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.018909    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="f34c7327-17c9-4085-8970-27dad43ee9be" containerName="istio-proxy"
Oct 30 20:57:27.018950 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.018920    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="f34c7327-17c9-4085-8970-27dad43ee9be" containerName="istio-proxy"
Oct 30 20:57:27.018950 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.018932    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="cce1c70d-cedf-494d-9abf-202a30a44263" containerName="payload-processing"
Oct 30 20:57:27.018950 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.018947    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bac9ec7-daeb-43cd-afb4-c8aca96f77a2" containerName="maas-api"
Oct 30 20:57:27.019117 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.018961    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="10329fd9-4ea0-41ac-a93b-6a0237b5f7c1" containerName="istio-proxy"
Oct 30 20:57:27.024438 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.024420    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x"
Oct 30 20:57:27.077648 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.077607    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:27.077850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.077620    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-9e0fb1-serving-cert\""
Oct 30 20:57:27.079304 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.079282    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7312d39c-e785-458a-9ce8-53ee254ac19c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x\" (UID: \"7312d39c-e785-458a-9ce8-53ee254ac19c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x"
Oct 30 20:57:27.079362 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.079345    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nztn4\" (UniqueName: \"kubernetes.io/projected/7312d39c-e785-458a-9ce8-53ee254ac19c-kube-api-access-nztn4\") pod \"maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x\" (UID: \"7312d39c-e785-458a-9ce8-53ee254ac19c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x"
Oct 30 20:57:27.079525 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.079503    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7312d39c-e785-458a-9ce8-53ee254ac19c-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x\" (UID: \"7312d39c-e785-458a-9ce8-53ee254ac19c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x"
Oct 30 20:57:27.079649 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.079630    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7312d39c-e785-458a-9ce8-53ee254ac19c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x\" (UID: \"7312d39c-e785-458a-9ce8-53ee254ac19c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x"
Oct 30 20:57:27.130596 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.130563    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x"]
Oct 30 20:57:27.181720 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.181658    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7312d39c-e785-458a-9ce8-53ee254ac19c-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x\" (UID: \"7312d39c-e785-458a-9ce8-53ee254ac19c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x"
Oct 30 20:57:27.182925 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.182899    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7312d39c-e785-458a-9ce8-53ee254ac19c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x\" (UID: \"7312d39c-e785-458a-9ce8-53ee254ac19c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x"
Oct 30 20:57:27.184634 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.183759    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7312d39c-e785-458a-9ce8-53ee254ac19c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x\" (UID: \"7312d39c-e785-458a-9ce8-53ee254ac19c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x"
Oct 30 20:57:27.184957 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.184938    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7312d39c-e785-458a-9ce8-53ee254ac19c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x\" (UID: \"7312d39c-e785-458a-9ce8-53ee254ac19c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x"
Oct 30 20:57:27.185239 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:57:27.185145    5071 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-9e0fb1-serving-cert: secret "maas-api-e2e-shared-b-w6-9e0fb1-serving-cert" not found
Oct 30 20:57:27.185404 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:57:27.185394    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7312d39c-e785-458a-9ce8-53ee254ac19c-maas-api-tls podName:7312d39c-e785-458a-9ce8-53ee254ac19c nodeName:}" failed. No retries permitted until 2026-10-30 20:57:27.685372787 +0000 UTC m=+1859.602394301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7312d39c-e785-458a-9ce8-53ee254ac19c-maas-api-tls") pod "maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x" (UID: "7312d39c-e785-458a-9ce8-53ee254ac19c") : secret "maas-api-e2e-shared-b-w6-9e0fb1-serving-cert" not found
Oct 30 20:57:27.185922 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.185804    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7312d39c-e785-458a-9ce8-53ee254ac19c-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x\" (UID: \"7312d39c-e785-458a-9ce8-53ee254ac19c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x"
Oct 30 20:57:27.186094 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.186068    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nztn4\" (UniqueName: \"kubernetes.io/projected/7312d39c-e785-458a-9ce8-53ee254ac19c-kube-api-access-nztn4\") pod \"maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x\" (UID: \"7312d39c-e785-458a-9ce8-53ee254ac19c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x"
Oct 30 20:57:27.244096 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.244056    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nztn4\" (UniqueName: \"kubernetes.io/projected/7312d39c-e785-458a-9ce8-53ee254ac19c-kube-api-access-nztn4\") pod \"maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x\" (UID: \"7312d39c-e785-458a-9ce8-53ee254ac19c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x"
Oct 30 20:57:27.246292 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.246262    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:27.415839 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.415793    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:27.585291 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.585254    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:27.704629 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.704547    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7312d39c-e785-458a-9ce8-53ee254ac19c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x\" (UID: \"7312d39c-e785-458a-9ce8-53ee254ac19c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x"
Oct 30 20:57:27.707104 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.707083    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7312d39c-e785-458a-9ce8-53ee254ac19c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x\" (UID: \"7312d39c-e785-458a-9ce8-53ee254ac19c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x"
Oct 30 20:57:27.940477 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:27.940441    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x"
Oct 30 20:57:28.016697 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:28.016654    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:28.204051 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:28.204020    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bac9ec7-daeb-43cd-afb4-c8aca96f77a2" path="/var/lib/kubelet/pods/6bac9ec7-daeb-43cd-afb4-c8aca96f77a2/volumes"
Oct 30 20:57:28.322719 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:28.322686    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x"]
Oct 30 20:57:28.322895 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:57:28.322815    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7312d39c_e785_458a_9ce8_53ee254ac19c.slice/crio-3378d0224d2052fa64d37cc4aeb5bcaed851c972e288839d183d68609e476128 WatchSource:0}: Error finding container 3378d0224d2052fa64d37cc4aeb5bcaed851c972e288839d183d68609e476128: Status 404 returned error can't find the container with id 3378d0224d2052fa64d37cc4aeb5bcaed851c972e288839d183d68609e476128
Oct 30 20:57:29.079783 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:29.079740    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x" event={"ID":"7312d39c-e785-458a-9ce8-53ee254ac19c","Type":"ContainerStarted","Data":"3378d0224d2052fa64d37cc4aeb5bcaed851c972e288839d183d68609e476128"}
Oct 30 20:57:30.089872 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:30.089809    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x" event={"ID":"7312d39c-e785-458a-9ce8-53ee254ac19c","Type":"ContainerStarted","Data":"9bdf859ce16951794bacae438d4c895f0fa7ece2720ddcfe1bdf0ef75013ac76"}
Oct 30 20:57:30.090339 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:30.089929    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x"
Oct 30 20:57:30.206814 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:30.205999    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x" podStartSLOduration=3.420014975 podStartE2EDuration="4.205981426s" totalImagesPullingTime="785.966451ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:30.205436586 +0000 UTC m=+1862.122458123" watchObservedRunningTime="2026-10-30 20:57:30.205981426 +0000 UTC m=+1862.123002963"
Oct 30 20:57:32.879612 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:32.879577    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:33.168094 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:33.167995    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:33.415094 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:33.415061    5071 scope.go:122] "RemoveContainer" containerID="0c99bcd96c6e941f2e680d9600249b536f2ad09013b8ecfd6d5d392a092f267a"
Oct 30 20:57:33.425205 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:33.425185    5071 scope.go:122] "RemoveContainer" containerID="080315184c09283122dcfc85e137f020b9ec79c72735e4f8653ad404796ae5b1"
Oct 30 20:57:33.434531 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:33.434506    5071 scope.go:122] "RemoveContainer" containerID="69cf5477e64e502404b807865ee3ca1517321b8520f1c70579a3ead966cb550a"
Oct 30 20:57:34.655089 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:34.655045    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:34.822754 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:34.822719    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:34.987685 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:34.987597    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:35.150127 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:35.150094    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"]
Oct 30 20:57:35.150328 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:35.150302    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc" podUID="690bc007-90bc-4e66-8c45-6425aaccc813" containerName="istio-proxy" containerID="cri-o://44e87a29af5b1c144a5b4e7a219f53c313a785d9b1606a09289212b698776052" gracePeriod=30
Oct 30 20:57:35.186286 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:35.186253    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:36.102765 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:36.102732    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x"
Oct 30 20:57:36.399622 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:36.399530    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:57:38.200913 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:38.200883    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:39.063279 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:39.063245    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:40.437060 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:40.436987    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:57:40.582309 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:40.582271    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/690bc007-90bc-4e66-8c45-6425aaccc813-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/690bc007-90bc-4e66-8c45-6425aaccc813-credential-socket\") pod \"690bc007-90bc-4e66-8c45-6425aaccc813\" (UID: \"690bc007-90bc-4e66-8c45-6425aaccc813\") "
Oct 30 20:57:40.582524 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:40.582345    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/690bc007-90bc-4e66-8c45-6425aaccc813-kube-api-access-27hwp\" (UniqueName: \"kubernetes.io/projected/690bc007-90bc-4e66-8c45-6425aaccc813-kube-api-access-27hwp\") pod \"690bc007-90bc-4e66-8c45-6425aaccc813\" (UID: \"690bc007-90bc-4e66-8c45-6425aaccc813\") "
Oct 30 20:57:40.582524 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:40.582380    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/690bc007-90bc-4e66-8c45-6425aaccc813-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/690bc007-90bc-4e66-8c45-6425aaccc813-workload-certs\") pod \"690bc007-90bc-4e66-8c45-6425aaccc813\" (UID: \"690bc007-90bc-4e66-8c45-6425aaccc813\") "
Oct 30 20:57:40.582524 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:40.582426    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/690bc007-90bc-4e66-8c45-6425aaccc813-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/690bc007-90bc-4e66-8c45-6425aaccc813-istiod-ca-cert\") pod \"690bc007-90bc-4e66-8c45-6425aaccc813\" (UID: \"690bc007-90bc-4e66-8c45-6425aaccc813\") "
Oct 30 20:57:40.582524 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:40.582457    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/690bc007-90bc-4e66-8c45-6425aaccc813-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/690bc007-90bc-4e66-8c45-6425aaccc813-istio-envoy\") pod \"690bc007-90bc-4e66-8c45-6425aaccc813\" (UID: \"690bc007-90bc-4e66-8c45-6425aaccc813\") "
Oct 30 20:57:40.582749 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:40.582593    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/690bc007-90bc-4e66-8c45-6425aaccc813-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/690bc007-90bc-4e66-8c45-6425aaccc813-workload-socket\") pod \"690bc007-90bc-4e66-8c45-6425aaccc813\" (UID: \"690bc007-90bc-4e66-8c45-6425aaccc813\") "
Oct 30 20:57:40.582749 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:40.582630    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/690bc007-90bc-4e66-8c45-6425aaccc813-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/690bc007-90bc-4e66-8c45-6425aaccc813-istio-podinfo\") pod \"690bc007-90bc-4e66-8c45-6425aaccc813\" (UID: \"690bc007-90bc-4e66-8c45-6425aaccc813\") "
Oct 30 20:57:40.582897 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:40.582756    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/690bc007-90bc-4e66-8c45-6425aaccc813-workload-certs" pod "690bc007-90bc-4e66-8c45-6425aaccc813" (UID: "690bc007-90bc-4e66-8c45-6425aaccc813"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:40.582897 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:40.582787    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/690bc007-90bc-4e66-8c45-6425aaccc813-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/690bc007-90bc-4e66-8c45-6425aaccc813-istio-data\") pod \"690bc007-90bc-4e66-8c45-6425aaccc813\" (UID: \"690bc007-90bc-4e66-8c45-6425aaccc813\") "
Oct 30 20:57:40.582897 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:40.582869    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/690bc007-90bc-4e66-8c45-6425aaccc813-istio-token\" (UniqueName: \"kubernetes.io/projected/690bc007-90bc-4e66-8c45-6425aaccc813-istio-token\") pod \"690bc007-90bc-4e66-8c45-6425aaccc813\" (UID: \"690bc007-90bc-4e66-8c45-6425aaccc813\") "
Oct 30 20:57:40.583158 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:40.583132    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/690bc007-90bc-4e66-8c45-6425aaccc813-credential-socket" pod "690bc007-90bc-4e66-8c45-6425aaccc813" (UID: "690bc007-90bc-4e66-8c45-6425aaccc813"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:40.583443 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:40.583420    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/690bc007-90bc-4e66-8c45-6425aaccc813-workload-socket" pod "690bc007-90bc-4e66-8c45-6425aaccc813" (UID: "690bc007-90bc-4e66-8c45-6425aaccc813"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:40.583569 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:40.583546    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/690bc007-90bc-4e66-8c45-6425aaccc813-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:40.583647 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:40.583577    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/690bc007-90bc-4e66-8c45-6425aaccc813-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:40.583795 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:40.583774    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/690bc007-90bc-4e66-8c45-6425aaccc813-istio-data" pod "690bc007-90bc-4e66-8c45-6425aaccc813" (UID: "690bc007-90bc-4e66-8c45-6425aaccc813"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:40.584191 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:40.584168    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/690bc007-90bc-4e66-8c45-6425aaccc813-istiod-ca-cert" pod "690bc007-90bc-4e66-8c45-6425aaccc813" (UID: "690bc007-90bc-4e66-8c45-6425aaccc813"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:40.586883 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:40.586857    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/690bc007-90bc-4e66-8c45-6425aaccc813-istio-envoy" pod "690bc007-90bc-4e66-8c45-6425aaccc813" (UID: "690bc007-90bc-4e66-8c45-6425aaccc813"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:40.587298 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:40.587274    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/690bc007-90bc-4e66-8c45-6425aaccc813-istio-podinfo" pod "690bc007-90bc-4e66-8c45-6425aaccc813" (UID: "690bc007-90bc-4e66-8c45-6425aaccc813"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:57:40.587968 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:40.587942    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/690bc007-90bc-4e66-8c45-6425aaccc813-kube-api-access-27hwp" pod "690bc007-90bc-4e66-8c45-6425aaccc813" (UID: "690bc007-90bc-4e66-8c45-6425aaccc813"). InnerVolumeSpecName "kube-api-access-27hwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:40.590071 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:40.590043    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/690bc007-90bc-4e66-8c45-6425aaccc813-istio-token" pod "690bc007-90bc-4e66-8c45-6425aaccc813" (UID: "690bc007-90bc-4e66-8c45-6425aaccc813"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:40.685377 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:40.685343    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27hwp\" (UniqueName: \"kubernetes.io/projected/690bc007-90bc-4e66-8c45-6425aaccc813-kube-api-access-27hwp\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:40.685377 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:40.685371    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/690bc007-90bc-4e66-8c45-6425aaccc813-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:40.685377 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:40.685383    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/690bc007-90bc-4e66-8c45-6425aaccc813-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:40.685610 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:40.685392    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/690bc007-90bc-4e66-8c45-6425aaccc813-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:40.685610 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:40.685402    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/690bc007-90bc-4e66-8c45-6425aaccc813-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:40.685610 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:40.685412    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/690bc007-90bc-4e66-8c45-6425aaccc813-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:40.685610 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:40.685421    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/690bc007-90bc-4e66-8c45-6425aaccc813-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:41.177463 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:41.177424    5071 generic.go:356] "Generic (PLEG): container finished" podID="690bc007-90bc-4e66-8c45-6425aaccc813" containerID="44e87a29af5b1c144a5b4e7a219f53c313a785d9b1606a09289212b698776052" exitCode=0
Oct 30 20:57:41.177651 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:41.177473    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"
Oct 30 20:57:41.177651 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:41.177504    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc" event={"ID":"690bc007-90bc-4e66-8c45-6425aaccc813","Type":"ContainerDied","Data":"44e87a29af5b1c144a5b4e7a219f53c313a785d9b1606a09289212b698776052"}
Oct 30 20:57:41.177651 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:41.177558    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc" event={"ID":"690bc007-90bc-4e66-8c45-6425aaccc813","Type":"ContainerDied","Data":"c4d07e4bf3cb6c3757e33a66491f25c16013a2a9c69063dfc7b30e2bb69a4d0c"}
Oct 30 20:57:41.177651 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:41.177573    5071 scope.go:122] "RemoveContainer" containerID="44e87a29af5b1c144a5b4e7a219f53c313a785d9b1606a09289212b698776052"
Oct 30 20:57:41.189378 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:41.189363    5071 scope.go:122] "RemoveContainer" containerID="fbba44203c5c4c37bf690ce7c7fe56f51c0c02777ad9ffb1ecad859af1f262a2"
Oct 30 20:57:41.199103 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:41.199082    5071 scope.go:122] "RemoveContainer" containerID="44e87a29af5b1c144a5b4e7a219f53c313a785d9b1606a09289212b698776052"
Oct 30 20:57:41.199353 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:57:41.199325    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44e87a29af5b1c144a5b4e7a219f53c313a785d9b1606a09289212b698776052\": container with ID starting with 44e87a29af5b1c144a5b4e7a219f53c313a785d9b1606a09289212b698776052 not found: ID does not exist" containerID="44e87a29af5b1c144a5b4e7a219f53c313a785d9b1606a09289212b698776052"
Oct 30 20:57:41.199411 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:41.199352    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44e87a29af5b1c144a5b4e7a219f53c313a785d9b1606a09289212b698776052"} err="failed to get container status \"44e87a29af5b1c144a5b4e7a219f53c313a785d9b1606a09289212b698776052\": rpc error: code = NotFound desc = could not find container \"44e87a29af5b1c144a5b4e7a219f53c313a785d9b1606a09289212b698776052\": container with ID starting with 44e87a29af5b1c144a5b4e7a219f53c313a785d9b1606a09289212b698776052 not found: ID does not exist"
Oct 30 20:57:41.199411 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:41.199372    5071 scope.go:122] "RemoveContainer" containerID="fbba44203c5c4c37bf690ce7c7fe56f51c0c02777ad9ffb1ecad859af1f262a2"
Oct 30 20:57:41.199586 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:57:41.199572    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbba44203c5c4c37bf690ce7c7fe56f51c0c02777ad9ffb1ecad859af1f262a2\": container with ID starting with fbba44203c5c4c37bf690ce7c7fe56f51c0c02777ad9ffb1ecad859af1f262a2 not found: ID does not exist" containerID="fbba44203c5c4c37bf690ce7c7fe56f51c0c02777ad9ffb1ecad859af1f262a2"
Oct 30 20:57:41.199621 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:41.199590    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbba44203c5c4c37bf690ce7c7fe56f51c0c02777ad9ffb1ecad859af1f262a2"} err="failed to get container status \"fbba44203c5c4c37bf690ce7c7fe56f51c0c02777ad9ffb1ecad859af1f262a2\": rpc error: code = NotFound desc = could not find container \"fbba44203c5c4c37bf690ce7c7fe56f51c0c02777ad9ffb1ecad859af1f262a2\": container with ID starting with fbba44203c5c4c37bf690ce7c7fe56f51c0c02777ad9ffb1ecad859af1f262a2 not found: ID does not exist"
Oct 30 20:57:41.396227 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:41.396195    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"]
Oct 30 20:57:41.446037 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:41.445955    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-0702ff-openshift-default-5d6966c459-8j9rc"]
Oct 30 20:57:42.202610 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:42.202574    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="690bc007-90bc-4e66-8c45-6425aaccc813" path="/var/lib/kubelet/pods/690bc007-90bc-4e66-8c45-6425aaccc813/volumes"
Oct 30 20:57:42.641583 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:42.641552    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r"]
Oct 30 20:57:42.642121 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:42.641796    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r" podUID="f53bc10d-bffe-418e-a223-15e324138b78" containerName="maas-api" containerID="cri-o://b2697c0891de36f8c66b74fccc9a058f126acb11a49226443313a971bd6ea9c6" gracePeriod=30
Oct 30 20:57:42.767178 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:42.767142    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:42.982616 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:42.982589    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r"
Oct 30 20:57:43.018501 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:43.018474    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f53bc10d-bffe-418e-a223-15e324138b78-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f53bc10d-bffe-418e-a223-15e324138b78-openshift-service-ca\") pod \"f53bc10d-bffe-418e-a223-15e324138b78\" (UID: \"f53bc10d-bffe-418e-a223-15e324138b78\") "
Oct 30 20:57:43.018730 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:43.018518    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f53bc10d-bffe-418e-a223-15e324138b78-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f53bc10d-bffe-418e-a223-15e324138b78-metrics-certs\") pod \"f53bc10d-bffe-418e-a223-15e324138b78\" (UID: \"f53bc10d-bffe-418e-a223-15e324138b78\") "
Oct 30 20:57:43.018730 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:43.018588    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f53bc10d-bffe-418e-a223-15e324138b78-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f53bc10d-bffe-418e-a223-15e324138b78-maas-api-tls\") pod \"f53bc10d-bffe-418e-a223-15e324138b78\" (UID: \"f53bc10d-bffe-418e-a223-15e324138b78\") "
Oct 30 20:57:43.018730 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:43.018637    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f53bc10d-bffe-418e-a223-15e324138b78-kube-api-access-9jhnb\" (UniqueName: \"kubernetes.io/projected/f53bc10d-bffe-418e-a223-15e324138b78-kube-api-access-9jhnb\") pod \"f53bc10d-bffe-418e-a223-15e324138b78\" (UID: \"f53bc10d-bffe-418e-a223-15e324138b78\") "
Oct 30 20:57:43.019301 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:43.018851    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f53bc10d-bffe-418e-a223-15e324138b78-openshift-service-ca" pod "f53bc10d-bffe-418e-a223-15e324138b78" (UID: "f53bc10d-bffe-418e-a223-15e324138b78"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:43.019301 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:43.019112    5071 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f53bc10d-bffe-418e-a223-15e324138b78-openshift-service-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:43.020989 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:43.020965    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f53bc10d-bffe-418e-a223-15e324138b78-maas-api-tls" pod "f53bc10d-bffe-418e-a223-15e324138b78" (UID: "f53bc10d-bffe-418e-a223-15e324138b78"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:57:43.021089 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:43.021037    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f53bc10d-bffe-418e-a223-15e324138b78-kube-api-access-9jhnb" pod "f53bc10d-bffe-418e-a223-15e324138b78" (UID: "f53bc10d-bffe-418e-a223-15e324138b78"). InnerVolumeSpecName "kube-api-access-9jhnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:43.021089 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:43.021055    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f53bc10d-bffe-418e-a223-15e324138b78-metrics-certs" pod "f53bc10d-bffe-418e-a223-15e324138b78" (UID: "f53bc10d-bffe-418e-a223-15e324138b78"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:57:43.126845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:43.120344    5071 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f53bc10d-bffe-418e-a223-15e324138b78-metrics-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:43.127122 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:43.127097    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f53bc10d-bffe-418e-a223-15e324138b78-maas-api-tls\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:43.127240 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:43.127224    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jhnb\" (UniqueName: \"kubernetes.io/projected/f53bc10d-bffe-418e-a223-15e324138b78-kube-api-access-9jhnb\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:43.195373 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:43.195338    5071 generic.go:356] "Generic (PLEG): container finished" podID="f53bc10d-bffe-418e-a223-15e324138b78" containerID="b2697c0891de36f8c66b74fccc9a058f126acb11a49226443313a971bd6ea9c6" exitCode=0
Oct 30 20:57:43.195541 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:43.195404    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r"
Oct 30 20:57:43.195541 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:43.195416    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r" event={"ID":"f53bc10d-bffe-418e-a223-15e324138b78","Type":"ContainerDied","Data":"b2697c0891de36f8c66b74fccc9a058f126acb11a49226443313a971bd6ea9c6"}
Oct 30 20:57:43.195541 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:43.195471    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r" event={"ID":"f53bc10d-bffe-418e-a223-15e324138b78","Type":"ContainerDied","Data":"9fb6bb191e76422e533cc3402278b1b8c552fe25ff306b3ef3bf8303c1446e31"}
Oct 30 20:57:43.195541 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:43.195482    5071 scope.go:122] "RemoveContainer" containerID="b2697c0891de36f8c66b74fccc9a058f126acb11a49226443313a971bd6ea9c6"
Oct 30 20:57:43.207569 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:43.207547    5071 scope.go:122] "RemoveContainer" containerID="b2697c0891de36f8c66b74fccc9a058f126acb11a49226443313a971bd6ea9c6"
Oct 30 20:57:43.207855 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:57:43.207815    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2697c0891de36f8c66b74fccc9a058f126acb11a49226443313a971bd6ea9c6\": container with ID starting with b2697c0891de36f8c66b74fccc9a058f126acb11a49226443313a971bd6ea9c6 not found: ID does not exist" containerID="b2697c0891de36f8c66b74fccc9a058f126acb11a49226443313a971bd6ea9c6"
Oct 30 20:57:43.207911 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:43.207863    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2697c0891de36f8c66b74fccc9a058f126acb11a49226443313a971bd6ea9c6"} err="failed to get container status \"b2697c0891de36f8c66b74fccc9a058f126acb11a49226443313a971bd6ea9c6\": rpc error: code = NotFound desc = could not find container \"b2697c0891de36f8c66b74fccc9a058f126acb11a49226443313a971bd6ea9c6\": container with ID starting with b2697c0891de36f8c66b74fccc9a058f126acb11a49226443313a971bd6ea9c6 not found: ID does not exist"
Oct 30 20:57:43.365420 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:43.365376    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r"]
Oct 30 20:57:43.368393 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:43.368371    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0c83a089-7d44cc85dd-tl27r"]
Oct 30 20:57:44.202955 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:44.202921    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f53bc10d-bffe-418e-a223-15e324138b78" path="/var/lib/kubelet/pods/f53bc10d-bffe-418e-a223-15e324138b78/volumes"
Oct 30 20:57:44.343192 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:44.343158    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:44.579589 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:44.579556    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:44.744122 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:44.744083    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:45.303606 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:45.303564    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:46.728738 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:57:46.728702    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:57:47.526606 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:47.526574    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-15199a-69b969cbfb-whtbl"]
Oct 30 20:57:47.634539 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:47.634503    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-15199a-79b8f8b875-m8l5v"]
Oct 30 20:57:47.945168 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:47.945137    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:48.124633 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:48.123756    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:50.080310 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:50.080272    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:52.028038 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:52.028000    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:52.244623 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:52.244572    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:52.330201 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:52.330167    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"]
Oct 30 20:57:52.330477 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:52.330440    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt" podUID="d204447d-7147-415b-abf8-853f68b51fec" containerName="istio-proxy" containerID="cri-o://3ac4564610203faa0abbfc3295a6b9971d34f0f348388302667edb1845e97926" gracePeriod=30
Oct 30 20:57:52.433379 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:52.433307    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:52.527315 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:52.527253    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-15199a-69b969cbfb-whtbl" podUID="9cb154a5-d317-4671-aa04-335b5a9060c7" containerName="payload-processing" containerID="cri-o://1076fb6ee6772a19b74dd9097ca02f5c02f8de41d661bdbab6999490c1800bd1" gracePeriod=25
Oct 30 20:57:52.635550 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:52.635454    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-15199a-79b8f8b875-m8l5v" podUID="281ee523-c301-4b62-a1ed-8d17e88bc88a" containerName="payload-pre-processing" containerID="cri-o://48a4c482cbf1b30f19ffe3735022eebb3f8a40a1cd1f13a29485162d3266ab33" gracePeriod=25
Oct 30 20:57:52.845728 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:52.845398    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-15199a-69b969cbfb-whtbl"
Oct 30 20:57:52.849126 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:52.848145    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:52.918003 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:52.917971    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cb154a5-d317-4671-aa04-335b5a9060c7-kube-api-access-6hffj\" (UniqueName: \"kubernetes.io/projected/9cb154a5-d317-4671-aa04-335b5a9060c7-kube-api-access-6hffj\") pod \"9cb154a5-d317-4671-aa04-335b5a9060c7\" (UID: \"9cb154a5-d317-4671-aa04-335b5a9060c7\") "
Oct 30 20:57:52.918158 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:52.918019    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cb154a5-d317-4671-aa04-335b5a9060c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cb154a5-d317-4671-aa04-335b5a9060c7-plugins-config-volume\") pod \"9cb154a5-d317-4671-aa04-335b5a9060c7\" (UID: \"9cb154a5-d317-4671-aa04-335b5a9060c7\") "
Oct 30 20:57:52.918876 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:52.918845    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cb154a5-d317-4671-aa04-335b5a9060c7-plugins-config-volume" pod "9cb154a5-d317-4671-aa04-335b5a9060c7" (UID: "9cb154a5-d317-4671-aa04-335b5a9060c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:52.920466 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:52.920438    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cb154a5-d317-4671-aa04-335b5a9060c7-kube-api-access-6hffj" pod "9cb154a5-d317-4671-aa04-335b5a9060c7" (UID: "9cb154a5-d317-4671-aa04-335b5a9060c7"). InnerVolumeSpecName "kube-api-access-6hffj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:52.941095 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:52.941070    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-15199a-79b8f8b875-m8l5v"
Oct 30 20:57:53.020035 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:53.019998    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/281ee523-c301-4b62-a1ed-8d17e88bc88a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281ee523-c301-4b62-a1ed-8d17e88bc88a-plugins-config-volume\") pod \"281ee523-c301-4b62-a1ed-8d17e88bc88a\" (UID: \"281ee523-c301-4b62-a1ed-8d17e88bc88a\") "
Oct 30 20:57:53.020236 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:53.020171    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/281ee523-c301-4b62-a1ed-8d17e88bc88a-kube-api-access-x22b4\" (UniqueName: \"kubernetes.io/projected/281ee523-c301-4b62-a1ed-8d17e88bc88a-kube-api-access-x22b4\") pod \"281ee523-c301-4b62-a1ed-8d17e88bc88a\" (UID: \"281ee523-c301-4b62-a1ed-8d17e88bc88a\") "
Oct 30 20:57:53.020433 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:53.020403    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/281ee523-c301-4b62-a1ed-8d17e88bc88a-plugins-config-volume" pod "281ee523-c301-4b62-a1ed-8d17e88bc88a" (UID: "281ee523-c301-4b62-a1ed-8d17e88bc88a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:53.020951 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:53.020923    5071 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281ee523-c301-4b62-a1ed-8d17e88bc88a-plugins-config-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:53.021060 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:53.020953    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hffj\" (UniqueName: \"kubernetes.io/projected/9cb154a5-d317-4671-aa04-335b5a9060c7-kube-api-access-6hffj\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:53.021060 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:53.020968    5071 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cb154a5-d317-4671-aa04-335b5a9060c7-plugins-config-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:53.022786 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:53.022759    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/281ee523-c301-4b62-a1ed-8d17e88bc88a-kube-api-access-x22b4" pod "281ee523-c301-4b62-a1ed-8d17e88bc88a" (UID: "281ee523-c301-4b62-a1ed-8d17e88bc88a"). InnerVolumeSpecName "kube-api-access-x22b4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:53.122526 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:53.122494    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x22b4\" (UniqueName: \"kubernetes.io/projected/281ee523-c301-4b62-a1ed-8d17e88bc88a-kube-api-access-x22b4\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:53.306632 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:53.306593    5071 generic.go:356] "Generic (PLEG): container finished" podID="9cb154a5-d317-4671-aa04-335b5a9060c7" containerID="1076fb6ee6772a19b74dd9097ca02f5c02f8de41d661bdbab6999490c1800bd1" exitCode=0
Oct 30 20:57:53.306856 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:53.306657    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-15199a-69b969cbfb-whtbl"
Oct 30 20:57:53.306856 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:53.306672    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-15199a-69b969cbfb-whtbl" event={"ID":"9cb154a5-d317-4671-aa04-335b5a9060c7","Type":"ContainerDied","Data":"1076fb6ee6772a19b74dd9097ca02f5c02f8de41d661bdbab6999490c1800bd1"}
Oct 30 20:57:53.306856 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:53.306734    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-15199a-69b969cbfb-whtbl" event={"ID":"9cb154a5-d317-4671-aa04-335b5a9060c7","Type":"ContainerDied","Data":"30acea1ca8e7308b508c2b5cf10c3132383a754ddcc65e0314b7aa8ace0386cd"}
Oct 30 20:57:53.306856 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:53.306759    5071 scope.go:122] "RemoveContainer" containerID="1076fb6ee6772a19b74dd9097ca02f5c02f8de41d661bdbab6999490c1800bd1"
Oct 30 20:57:53.308338 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:53.308312    5071 generic.go:356] "Generic (PLEG): container finished" podID="281ee523-c301-4b62-a1ed-8d17e88bc88a" containerID="48a4c482cbf1b30f19ffe3735022eebb3f8a40a1cd1f13a29485162d3266ab33" exitCode=0
Oct 30 20:57:53.308464 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:53.308356    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-15199a-79b8f8b875-m8l5v"
Oct 30 20:57:53.308464 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:53.308356    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-15199a-79b8f8b875-m8l5v" event={"ID":"281ee523-c301-4b62-a1ed-8d17e88bc88a","Type":"ContainerDied","Data":"48a4c482cbf1b30f19ffe3735022eebb3f8a40a1cd1f13a29485162d3266ab33"}
Oct 30 20:57:53.308464 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:53.308407    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-15199a-79b8f8b875-m8l5v" event={"ID":"281ee523-c301-4b62-a1ed-8d17e88bc88a","Type":"ContainerDied","Data":"11c70064bc3675ea91e11af11a485e9dd288c414912b6db89da5a3d56860c3ba"}
Oct 30 20:57:53.319729 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:53.319712    5071 scope.go:122] "RemoveContainer" containerID="1076fb6ee6772a19b74dd9097ca02f5c02f8de41d661bdbab6999490c1800bd1"
Oct 30 20:57:53.320061 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:57:53.320041    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1076fb6ee6772a19b74dd9097ca02f5c02f8de41d661bdbab6999490c1800bd1\": container with ID starting with 1076fb6ee6772a19b74dd9097ca02f5c02f8de41d661bdbab6999490c1800bd1 not found: ID does not exist" containerID="1076fb6ee6772a19b74dd9097ca02f5c02f8de41d661bdbab6999490c1800bd1"
Oct 30 20:57:53.320176 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:53.320069    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1076fb6ee6772a19b74dd9097ca02f5c02f8de41d661bdbab6999490c1800bd1"} err="failed to get container status \"1076fb6ee6772a19b74dd9097ca02f5c02f8de41d661bdbab6999490c1800bd1\": rpc error: code = NotFound desc = could not find container \"1076fb6ee6772a19b74dd9097ca02f5c02f8de41d661bdbab6999490c1800bd1\": container with ID starting with 1076fb6ee6772a19b74dd9097ca02f5c02f8de41d661bdbab6999490c1800bd1 not found: ID does not exist"
Oct 30 20:57:53.320176 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:53.320085    5071 scope.go:122] "RemoveContainer" containerID="48a4c482cbf1b30f19ffe3735022eebb3f8a40a1cd1f13a29485162d3266ab33"
Oct 30 20:57:53.330693 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:53.330669    5071 scope.go:122] "RemoveContainer" containerID="48a4c482cbf1b30f19ffe3735022eebb3f8a40a1cd1f13a29485162d3266ab33"
Oct 30 20:57:53.331018 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:57:53.330995    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48a4c482cbf1b30f19ffe3735022eebb3f8a40a1cd1f13a29485162d3266ab33\": container with ID starting with 48a4c482cbf1b30f19ffe3735022eebb3f8a40a1cd1f13a29485162d3266ab33 not found: ID does not exist" containerID="48a4c482cbf1b30f19ffe3735022eebb3f8a40a1cd1f13a29485162d3266ab33"
Oct 30 20:57:53.331092 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:53.331027    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48a4c482cbf1b30f19ffe3735022eebb3f8a40a1cd1f13a29485162d3266ab33"} err="failed to get container status \"48a4c482cbf1b30f19ffe3735022eebb3f8a40a1cd1f13a29485162d3266ab33\": rpc error: code = NotFound desc = could not find container \"48a4c482cbf1b30f19ffe3735022eebb3f8a40a1cd1f13a29485162d3266ab33\": container with ID starting with 48a4c482cbf1b30f19ffe3735022eebb3f8a40a1cd1f13a29485162d3266ab33 not found: ID does not exist"
Oct 30 20:57:53.474224 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:53.474191    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-15199a-69b969cbfb-whtbl"]
Oct 30 20:57:53.523748 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:53.523708    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-15199a-69b969cbfb-whtbl"]
Oct 30 20:57:53.633610 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:53.633528    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-15199a-79b8f8b875-m8l5v"]
Oct 30 20:57:53.640739 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:53.640711    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-15199a-79b8f8b875-m8l5v"]
Oct 30 20:57:54.203040 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:54.203006    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="281ee523-c301-4b62-a1ed-8d17e88bc88a" path="/var/lib/kubelet/pods/281ee523-c301-4b62-a1ed-8d17e88bc88a/volumes"
Oct 30 20:57:54.203459 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:54.203445    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cb154a5-d317-4671-aa04-335b5a9060c7" path="/var/lib/kubelet/pods/9cb154a5-d317-4671-aa04-335b5a9060c7/volumes"
Oct 30 20:57:54.987948 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:54.987917    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f"]
Oct 30 20:57:54.988577 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:54.988563    5071 cpu_manager.go:414] "containerMap: removing container" podUID="f34c7327-17c9-4085-8970-27dad43ee9be" containerName="istio-proxy"
Oct 30 20:57:54.988635 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:54.988578    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f34c7327-17c9-4085-8970-27dad43ee9be" containerName="istio-proxy"
Oct 30 20:57:54.988635 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:54.988595    5071 cpu_manager.go:414] "containerMap: removing container" podUID="9cb154a5-d317-4671-aa04-335b5a9060c7" containerName="payload-processing"
Oct 30 20:57:54.988635 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:54.988601    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cb154a5-d317-4671-aa04-335b5a9060c7" containerName="payload-processing"
Oct 30 20:57:54.988635 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:54.988614    5071 cpu_manager.go:414] "containerMap: removing container" podUID="690bc007-90bc-4e66-8c45-6425aaccc813" containerName="istio-proxy"
Oct 30 20:57:54.988635 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:54.988619    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="690bc007-90bc-4e66-8c45-6425aaccc813" containerName="istio-proxy"
Oct 30 20:57:54.988635 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:54.988626    5071 cpu_manager.go:414] "containerMap: removing container" podUID="f53bc10d-bffe-418e-a223-15e324138b78" containerName="maas-api"
Oct 30 20:57:54.988635 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:54.988631    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f53bc10d-bffe-418e-a223-15e324138b78" containerName="maas-api"
Oct 30 20:57:54.988884 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:54.988643    5071 cpu_manager.go:414] "containerMap: removing container" podUID="281ee523-c301-4b62-a1ed-8d17e88bc88a" containerName="payload-pre-processing"
Oct 30 20:57:54.988884 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:54.988648    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="281ee523-c301-4b62-a1ed-8d17e88bc88a" containerName="payload-pre-processing"
Oct 30 20:57:54.988884 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:54.988678    5071 cpu_manager.go:414] "containerMap: removing container" podUID="690bc007-90bc-4e66-8c45-6425aaccc813" containerName="istio-proxy"
Oct 30 20:57:54.988884 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:54.988683    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="690bc007-90bc-4e66-8c45-6425aaccc813" containerName="istio-proxy"
Oct 30 20:57:54.988884 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:54.988773    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="690bc007-90bc-4e66-8c45-6425aaccc813" containerName="istio-proxy"
Oct 30 20:57:54.988884 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:54.988780    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="f53bc10d-bffe-418e-a223-15e324138b78" containerName="maas-api"
Oct 30 20:57:54.988884 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:54.988787    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="281ee523-c301-4b62-a1ed-8d17e88bc88a" containerName="payload-pre-processing"
Oct 30 20:57:54.988884 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:54.988798    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="690bc007-90bc-4e66-8c45-6425aaccc813" containerName="istio-proxy"
Oct 30 20:57:54.988884 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:54.988806    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cb154a5-d317-4671-aa04-335b5a9060c7" containerName="payload-processing"
Oct 30 20:57:54.994024 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:54.994001    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f"
Oct 30 20:57:55.096678 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:55.096644    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-19176d-serving-cert\""
Oct 30 20:57:55.100033 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:55.100008    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f"]
Oct 30 20:57:55.149055 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:55.149027    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a7dd791-abaf-4b61-8fbf-25175613fb66-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f\" (UID: \"0a7dd791-abaf-4b61-8fbf-25175613fb66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f"
Oct 30 20:57:55.149197 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:55.149160    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfcrd\" (UniqueName: \"kubernetes.io/projected/0a7dd791-abaf-4b61-8fbf-25175613fb66-kube-api-access-vfcrd\") pod \"maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f\" (UID: \"0a7dd791-abaf-4b61-8fbf-25175613fb66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f"
Oct 30 20:57:55.149244 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:55.149218    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a7dd791-abaf-4b61-8fbf-25175613fb66-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f\" (UID: \"0a7dd791-abaf-4b61-8fbf-25175613fb66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f"
Oct 30 20:57:55.149372 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:55.149351    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a7dd791-abaf-4b61-8fbf-25175613fb66-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f\" (UID: \"0a7dd791-abaf-4b61-8fbf-25175613fb66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f"
Oct 30 20:57:55.250454 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:55.250374    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a7dd791-abaf-4b61-8fbf-25175613fb66-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f\" (UID: \"0a7dd791-abaf-4b61-8fbf-25175613fb66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f"
Oct 30 20:57:55.250885 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:55.250474    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a7dd791-abaf-4b61-8fbf-25175613fb66-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f\" (UID: \"0a7dd791-abaf-4b61-8fbf-25175613fb66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f"
Oct 30 20:57:55.250885 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:55.250568    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfcrd\" (UniqueName: \"kubernetes.io/projected/0a7dd791-abaf-4b61-8fbf-25175613fb66-kube-api-access-vfcrd\") pod \"maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f\" (UID: \"0a7dd791-abaf-4b61-8fbf-25175613fb66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f"
Oct 30 20:57:55.250885 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:57:55.250626    5071 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-19176d-serving-cert: secret "maas-api-e2e-shared-b-w4-19176d-serving-cert" not found
Oct 30 20:57:55.250885 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:57:55.250708    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a7dd791-abaf-4b61-8fbf-25175613fb66-maas-api-tls podName:0a7dd791-abaf-4b61-8fbf-25175613fb66 nodeName:}" failed. No retries permitted until 2026-10-30 20:57:55.750692321 +0000 UTC m=+1887.667713839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0a7dd791-abaf-4b61-8fbf-25175613fb66-maas-api-tls") pod "maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f" (UID: "0a7dd791-abaf-4b61-8fbf-25175613fb66") : secret "maas-api-e2e-shared-b-w4-19176d-serving-cert" not found
Oct 30 20:57:55.250885 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:55.250630    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a7dd791-abaf-4b61-8fbf-25175613fb66-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f\" (UID: \"0a7dd791-abaf-4b61-8fbf-25175613fb66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f"
Oct 30 20:57:55.251231 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:55.251214    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a7dd791-abaf-4b61-8fbf-25175613fb66-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f\" (UID: \"0a7dd791-abaf-4b61-8fbf-25175613fb66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f"
Oct 30 20:57:55.253002 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:55.252978    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a7dd791-abaf-4b61-8fbf-25175613fb66-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f\" (UID: \"0a7dd791-abaf-4b61-8fbf-25175613fb66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f"
Oct 30 20:57:55.280411 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:55.280376    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:55.315374 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:55.315340    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfcrd\" (UniqueName: \"kubernetes.io/projected/0a7dd791-abaf-4b61-8fbf-25175613fb66-kube-api-access-vfcrd\") pod \"maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f\" (UID: \"0a7dd791-abaf-4b61-8fbf-25175613fb66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f"
Oct 30 20:57:55.503534 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:55.503444    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:55.656460 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:55.656422    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:55.803806 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:55.803768    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a7dd791-abaf-4b61-8fbf-25175613fb66-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f\" (UID: \"0a7dd791-abaf-4b61-8fbf-25175613fb66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f"
Oct 30 20:57:55.806540 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:55.806515    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a7dd791-abaf-4b61-8fbf-25175613fb66-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f\" (UID: \"0a7dd791-abaf-4b61-8fbf-25175613fb66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f"
Oct 30 20:57:55.909430 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:55.909398    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f"
Oct 30 20:57:56.296473 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:56.296447    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f"]
Oct 30 20:57:56.298682 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:57:56.298656    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a7dd791_abaf_4b61_8fbf_25175613fb66.slice/crio-4909232f32052ac2af0e6e47c67f8375dbeb5d0fa6f35ee2399346f064c3626a WatchSource:0}: Error finding container 4909232f32052ac2af0e6e47c67f8375dbeb5d0fa6f35ee2399346f064c3626a: Status 404 returned error can't find the container with id 4909232f32052ac2af0e6e47c67f8375dbeb5d0fa6f35ee2399346f064c3626a
Oct 30 20:57:56.300500 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:56.300481    5071 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:57:56.336552 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:56.336524    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f" event={"ID":"0a7dd791-abaf-4b61-8fbf-25175613fb66","Type":"ContainerStarted","Data":"4909232f32052ac2af0e6e47c67f8375dbeb5d0fa6f35ee2399346f064c3626a"}
Oct 30 20:57:56.622091 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:56.622008    5071 patch_prober.go:29] interesting pod/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:57:56.622091 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:56.622069    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt" podUID="d204447d-7147-415b-abf8-853f68b51fec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:57:57.403733 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:57.403614    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f" event={"ID":"0a7dd791-abaf-4b61-8fbf-25175613fb66","Type":"ContainerStarted","Data":"33a8247b2077fe7510522ed1ad751178f638b658b00e01eb27e52a9801fa4f85"}
Oct 30 20:57:57.404272 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:57.404020    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f"
Oct 30 20:57:57.531480 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:57.529679    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f" podStartSLOduration=2.694575689 podStartE2EDuration="3.529658063s" totalImagesPullingTime="835.082374ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:57.525977402 +0000 UTC m=+1889.442998942" watchObservedRunningTime="2026-10-30 20:57:57.529658063 +0000 UTC m=+1889.446679600"
Oct 30 20:57:57.638663 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:57.638625    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:57:57.735643 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:57.735552    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d204447d-7147-415b-abf8-853f68b51fec-kube-api-access-7lmc2\" (UniqueName: \"kubernetes.io/projected/d204447d-7147-415b-abf8-853f68b51fec-kube-api-access-7lmc2\") pod \"d204447d-7147-415b-abf8-853f68b51fec\" (UID: \"d204447d-7147-415b-abf8-853f68b51fec\") "
Oct 30 20:57:57.735643 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:57.735592    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d204447d-7147-415b-abf8-853f68b51fec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d204447d-7147-415b-abf8-853f68b51fec-credential-socket\") pod \"d204447d-7147-415b-abf8-853f68b51fec\" (UID: \"d204447d-7147-415b-abf8-853f68b51fec\") "
Oct 30 20:57:57.735643 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:57.735628    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d204447d-7147-415b-abf8-853f68b51fec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d204447d-7147-415b-abf8-853f68b51fec-istio-data\") pod \"d204447d-7147-415b-abf8-853f68b51fec\" (UID: \"d204447d-7147-415b-abf8-853f68b51fec\") "
Oct 30 20:57:57.735984 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:57.735667    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d204447d-7147-415b-abf8-853f68b51fec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d204447d-7147-415b-abf8-853f68b51fec-workload-certs\") pod \"d204447d-7147-415b-abf8-853f68b51fec\" (UID: \"d204447d-7147-415b-abf8-853f68b51fec\") "
Oct 30 20:57:57.735984 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:57.735704    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d204447d-7147-415b-abf8-853f68b51fec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d204447d-7147-415b-abf8-853f68b51fec-workload-socket\") pod \"d204447d-7147-415b-abf8-853f68b51fec\" (UID: \"d204447d-7147-415b-abf8-853f68b51fec\") "
Oct 30 20:57:57.735984 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:57.735804    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d204447d-7147-415b-abf8-853f68b51fec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d204447d-7147-415b-abf8-853f68b51fec-istiod-ca-cert\") pod \"d204447d-7147-415b-abf8-853f68b51fec\" (UID: \"d204447d-7147-415b-abf8-853f68b51fec\") "
Oct 30 20:57:57.735984 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:57.735853    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d204447d-7147-415b-abf8-853f68b51fec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d204447d-7147-415b-abf8-853f68b51fec-istio-envoy\") pod \"d204447d-7147-415b-abf8-853f68b51fec\" (UID: \"d204447d-7147-415b-abf8-853f68b51fec\") "
Oct 30 20:57:57.736196 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:57.736005    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d204447d-7147-415b-abf8-853f68b51fec-istio-token\" (UniqueName: \"kubernetes.io/projected/d204447d-7147-415b-abf8-853f68b51fec-istio-token\") pod \"d204447d-7147-415b-abf8-853f68b51fec\" (UID: \"d204447d-7147-415b-abf8-853f68b51fec\") "
Oct 30 20:57:57.736196 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:57.736009    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d204447d-7147-415b-abf8-853f68b51fec-istio-data" pod "d204447d-7147-415b-abf8-853f68b51fec" (UID: "d204447d-7147-415b-abf8-853f68b51fec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:57.736196 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:57.736020    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d204447d-7147-415b-abf8-853f68b51fec-credential-socket" pod "d204447d-7147-415b-abf8-853f68b51fec" (UID: "d204447d-7147-415b-abf8-853f68b51fec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:57.736196 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:57.736051    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d204447d-7147-415b-abf8-853f68b51fec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d204447d-7147-415b-abf8-853f68b51fec-istio-podinfo\") pod \"d204447d-7147-415b-abf8-853f68b51fec\" (UID: \"d204447d-7147-415b-abf8-853f68b51fec\") "
Oct 30 20:57:57.736469 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:57.736445    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d204447d-7147-415b-abf8-853f68b51fec-istiod-ca-cert" pod "d204447d-7147-415b-abf8-853f68b51fec" (UID: "d204447d-7147-415b-abf8-853f68b51fec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:57.736568 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:57.736544    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d204447d-7147-415b-abf8-853f68b51fec-workload-certs" pod "d204447d-7147-415b-abf8-853f68b51fec" (UID: "d204447d-7147-415b-abf8-853f68b51fec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:57.736745 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:57.736695    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d204447d-7147-415b-abf8-853f68b51fec-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:57.736915 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:57.736900    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d204447d-7147-415b-abf8-853f68b51fec-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:57.737044 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:57.737030    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d204447d-7147-415b-abf8-853f68b51fec-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:57.737139 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:57.736787    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d204447d-7147-415b-abf8-853f68b51fec-workload-socket" pod "d204447d-7147-415b-abf8-853f68b51fec" (UID: "d204447d-7147-415b-abf8-853f68b51fec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:57.738678 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:57.738635    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d204447d-7147-415b-abf8-853f68b51fec-istio-envoy" pod "d204447d-7147-415b-abf8-853f68b51fec" (UID: "d204447d-7147-415b-abf8-853f68b51fec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:57.738960 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:57.738935    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d204447d-7147-415b-abf8-853f68b51fec-istio-podinfo" pod "d204447d-7147-415b-abf8-853f68b51fec" (UID: "d204447d-7147-415b-abf8-853f68b51fec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:57:57.739059 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:57.738959    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d204447d-7147-415b-abf8-853f68b51fec-kube-api-access-7lmc2" pod "d204447d-7147-415b-abf8-853f68b51fec" (UID: "d204447d-7147-415b-abf8-853f68b51fec"). InnerVolumeSpecName "kube-api-access-7lmc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:57.739736 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:57.739710    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d204447d-7147-415b-abf8-853f68b51fec-istio-token" pod "d204447d-7147-415b-abf8-853f68b51fec" (UID: "d204447d-7147-415b-abf8-853f68b51fec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:57.838927 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:57.838886    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d204447d-7147-415b-abf8-853f68b51fec-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:57.838927 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:57.838927    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lmc2\" (UniqueName: \"kubernetes.io/projected/d204447d-7147-415b-abf8-853f68b51fec-kube-api-access-7lmc2\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:57.839134 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:57.838944    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d204447d-7147-415b-abf8-853f68b51fec-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:57.839134 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:57.838958    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d204447d-7147-415b-abf8-853f68b51fec-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:57.839134 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:57.838972    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d204447d-7147-415b-abf8-853f68b51fec-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:57.839134 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:57.838984    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d204447d-7147-415b-abf8-853f68b51fec-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:57.934272 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:57.934232    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:57:58.412703 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:58.412665    5071 generic.go:356] "Generic (PLEG): container finished" podID="d204447d-7147-415b-abf8-853f68b51fec" containerID="3ac4564610203faa0abbfc3295a6b9971d34f0f348388302667edb1845e97926" exitCode=0
Oct 30 20:57:58.413218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:58.412718    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"
Oct 30 20:57:58.413218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:58.412769    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt" event={"ID":"d204447d-7147-415b-abf8-853f68b51fec","Type":"ContainerDied","Data":"3ac4564610203faa0abbfc3295a6b9971d34f0f348388302667edb1845e97926"}
Oct 30 20:57:58.413218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:58.412841    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt" event={"ID":"d204447d-7147-415b-abf8-853f68b51fec","Type":"ContainerDied","Data":"bf4248d05791d47248030924253ecbd8fdad4569eedc56630c85abc424bdf52c"}
Oct 30 20:57:58.413218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:58.412875    5071 scope.go:122] "RemoveContainer" containerID="3ac4564610203faa0abbfc3295a6b9971d34f0f348388302667edb1845e97926"
Oct 30 20:57:58.512337 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:58.512310    5071 scope.go:122] "RemoveContainer" containerID="7da7aee651df96f9cb6a880e8f7d572a730af37552f32977d48c2c16a6c2edaa"
Oct 30 20:57:58.529219 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:58.529102    5071 scope.go:122] "RemoveContainer" containerID="3ac4564610203faa0abbfc3295a6b9971d34f0f348388302667edb1845e97926"
Oct 30 20:57:58.529536 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:57:58.529498    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ac4564610203faa0abbfc3295a6b9971d34f0f348388302667edb1845e97926\": container with ID starting with 3ac4564610203faa0abbfc3295a6b9971d34f0f348388302667edb1845e97926 not found: ID does not exist" containerID="3ac4564610203faa0abbfc3295a6b9971d34f0f348388302667edb1845e97926"
Oct 30 20:57:58.529667 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:58.529535    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ac4564610203faa0abbfc3295a6b9971d34f0f348388302667edb1845e97926"} err="failed to get container status \"3ac4564610203faa0abbfc3295a6b9971d34f0f348388302667edb1845e97926\": rpc error: code = NotFound desc = could not find container \"3ac4564610203faa0abbfc3295a6b9971d34f0f348388302667edb1845e97926\": container with ID starting with 3ac4564610203faa0abbfc3295a6b9971d34f0f348388302667edb1845e97926 not found: ID does not exist"
Oct 30 20:57:58.529667 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:58.529560    5071 scope.go:122] "RemoveContainer" containerID="7da7aee651df96f9cb6a880e8f7d572a730af37552f32977d48c2c16a6c2edaa"
Oct 30 20:57:58.530195 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:57:58.530097    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7da7aee651df96f9cb6a880e8f7d572a730af37552f32977d48c2c16a6c2edaa\": container with ID starting with 7da7aee651df96f9cb6a880e8f7d572a730af37552f32977d48c2c16a6c2edaa not found: ID does not exist" containerID="7da7aee651df96f9cb6a880e8f7d572a730af37552f32977d48c2c16a6c2edaa"
Oct 30 20:57:58.530195 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:58.530129    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7da7aee651df96f9cb6a880e8f7d572a730af37552f32977d48c2c16a6c2edaa"} err="failed to get container status \"7da7aee651df96f9cb6a880e8f7d572a730af37552f32977d48c2c16a6c2edaa\": rpc error: code = NotFound desc = could not find container \"7da7aee651df96f9cb6a880e8f7d572a730af37552f32977d48c2c16a6c2edaa\": container with ID starting with 7da7aee651df96f9cb6a880e8f7d572a730af37552f32977d48c2c16a6c2edaa not found: ID does not exist"
Oct 30 20:57:58.602706 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:58.602680    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"]
Oct 30 20:57:58.651946 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:57:58.651920    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-0c83a089-openshift-default-85d6c697f7-fwjzt"]
Oct 30 20:58:00.202936 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:00.202903    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d204447d-7147-415b-abf8-853f68b51fec" path="/var/lib/kubelet/pods/d204447d-7147-415b-abf8-853f68b51fec/volumes"
Oct 30 20:58:00.690311 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:00.690277    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:03.419860 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:03.419810    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f"
Oct 30 20:58:03.822596 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:03.822563    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:58:04.905375 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:04.905337    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc"]
Oct 30 20:58:04.905923 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:04.905555    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc" podUID="a458b472-4aae-4dcc-87c9-cadd0a317061" containerName="maas-api" containerID="cri-o://7215e93e640b303c5a0ea469fa080c2216dd684f50b8bb5cacbb064913f49a3f" gracePeriod=30
Oct 30 20:58:05.027248 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:05.027181    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:05.222045 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:05.221961    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f6ebf5-69984c68d7-ls9bm"]
Oct 30 20:58:05.396038 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:05.388624    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc"
Oct 30 20:58:05.419269 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:05.415532    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a458b472-4aae-4dcc-87c9-cadd0a317061-kube-api-access-5wkb6\" (UniqueName: \"kubernetes.io/projected/a458b472-4aae-4dcc-87c9-cadd0a317061-kube-api-access-5wkb6\") pod \"a458b472-4aae-4dcc-87c9-cadd0a317061\" (UID: \"a458b472-4aae-4dcc-87c9-cadd0a317061\") "
Oct 30 20:58:05.419269 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:05.415680    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a458b472-4aae-4dcc-87c9-cadd0a317061-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a458b472-4aae-4dcc-87c9-cadd0a317061-openshift-service-ca\") pod \"a458b472-4aae-4dcc-87c9-cadd0a317061\" (UID: \"a458b472-4aae-4dcc-87c9-cadd0a317061\") "
Oct 30 20:58:05.419269 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:05.415758    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a458b472-4aae-4dcc-87c9-cadd0a317061-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a458b472-4aae-4dcc-87c9-cadd0a317061-maas-api-tls\") pod \"a458b472-4aae-4dcc-87c9-cadd0a317061\" (UID: \"a458b472-4aae-4dcc-87c9-cadd0a317061\") "
Oct 30 20:58:05.419269 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:05.415784    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a458b472-4aae-4dcc-87c9-cadd0a317061-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a458b472-4aae-4dcc-87c9-cadd0a317061-metrics-certs\") pod \"a458b472-4aae-4dcc-87c9-cadd0a317061\" (UID: \"a458b472-4aae-4dcc-87c9-cadd0a317061\") "
Oct 30 20:58:05.421068 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:05.421026    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a458b472-4aae-4dcc-87c9-cadd0a317061-openshift-service-ca" pod "a458b472-4aae-4dcc-87c9-cadd0a317061" (UID: "a458b472-4aae-4dcc-87c9-cadd0a317061"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:58:05.437906 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:05.428972    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a458b472-4aae-4dcc-87c9-cadd0a317061-metrics-certs" pod "a458b472-4aae-4dcc-87c9-cadd0a317061" (UID: "a458b472-4aae-4dcc-87c9-cadd0a317061"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:58:05.437906 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:05.429565    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a458b472-4aae-4dcc-87c9-cadd0a317061-kube-api-access-5wkb6" pod "a458b472-4aae-4dcc-87c9-cadd0a317061" (UID: "a458b472-4aae-4dcc-87c9-cadd0a317061"). InnerVolumeSpecName "kube-api-access-5wkb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:58:05.442771 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:05.442724    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a458b472-4aae-4dcc-87c9-cadd0a317061-maas-api-tls" pod "a458b472-4aae-4dcc-87c9-cadd0a317061" (UID: "a458b472-4aae-4dcc-87c9-cadd0a317061"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:58:05.446044 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:05.444805    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:05.527169 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:05.527082    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wkb6\" (UniqueName: \"kubernetes.io/projected/a458b472-4aae-4dcc-87c9-cadd0a317061-kube-api-access-5wkb6\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:05.527401 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:05.527378    5071 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a458b472-4aae-4dcc-87c9-cadd0a317061-openshift-service-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:05.527546 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:05.527532    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a458b472-4aae-4dcc-87c9-cadd0a317061-maas-api-tls\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:05.527646 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:05.527634    5071 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a458b472-4aae-4dcc-87c9-cadd0a317061-metrics-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:05.610320 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:05.610151    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:05.613123 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:05.613097    5071 generic.go:356] "Generic (PLEG): container finished" podID="a458b472-4aae-4dcc-87c9-cadd0a317061" containerID="7215e93e640b303c5a0ea469fa080c2216dd684f50b8bb5cacbb064913f49a3f" exitCode=0
Oct 30 20:58:05.613519 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:05.613499    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc" event={"ID":"a458b472-4aae-4dcc-87c9-cadd0a317061","Type":"ContainerDied","Data":"7215e93e640b303c5a0ea469fa080c2216dd684f50b8bb5cacbb064913f49a3f"}
Oct 30 20:58:05.613665 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:05.613651    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc" event={"ID":"a458b472-4aae-4dcc-87c9-cadd0a317061","Type":"ContainerDied","Data":"bd6fe7760ff1f19aa4791edeb2bd39f202da4c514a7e476db4876fb5902b59c7"}
Oct 30 20:58:05.613768 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:05.613757    5071 scope.go:122] "RemoveContainer" containerID="7215e93e640b303c5a0ea469fa080c2216dd684f50b8bb5cacbb064913f49a3f"
Oct 30 20:58:05.614057 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:05.614041    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc"
Oct 30 20:58:05.729850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:05.729804    5071 scope.go:122] "RemoveContainer" containerID="7215e93e640b303c5a0ea469fa080c2216dd684f50b8bb5cacbb064913f49a3f"
Oct 30 20:58:05.730481 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:58:05.730459    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7215e93e640b303c5a0ea469fa080c2216dd684f50b8bb5cacbb064913f49a3f\": container with ID starting with 7215e93e640b303c5a0ea469fa080c2216dd684f50b8bb5cacbb064913f49a3f not found: ID does not exist" containerID="7215e93e640b303c5a0ea469fa080c2216dd684f50b8bb5cacbb064913f49a3f"
Oct 30 20:58:05.730671 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:05.730633    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7215e93e640b303c5a0ea469fa080c2216dd684f50b8bb5cacbb064913f49a3f"} err="failed to get container status \"7215e93e640b303c5a0ea469fa080c2216dd684f50b8bb5cacbb064913f49a3f\": rpc error: code = NotFound desc = could not find container \"7215e93e640b303c5a0ea469fa080c2216dd684f50b8bb5cacbb064913f49a3f\": container with ID starting with 7215e93e640b303c5a0ea469fa080c2216dd684f50b8bb5cacbb064913f49a3f not found: ID does not exist"
Oct 30 20:58:05.856546 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:05.856510    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc"]
Oct 30 20:58:05.861430 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:05.861405    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-aaf669-7f7465567b-ktprc"]
Oct 30 20:58:06.216848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:06.216723    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a458b472-4aae-4dcc-87c9-cadd0a317061" path="/var/lib/kubelet/pods/a458b472-4aae-4dcc-87c9-cadd0a317061/volumes"
Oct 30 20:58:06.692535 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:06.692459    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:07.053090 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:07.053050    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:07.216685 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:07.216644    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:07.683551 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:07.683516    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:10.222905 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:10.222854    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f6ebf5-69984c68d7-ls9bm" podUID="8bbebe9e-cd3b-49d7-8048-bde0a0faafa2" containerName="payload-pre-processing" containerID="cri-o://934149a7dc3af11fe6a99e9e2eba31230ffc4ef52bc894929cf10393cfe256b7" gracePeriod=25
Oct 30 20:58:10.553378 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:10.553338    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f6ebf5-69984c68d7-ls9bm" podUID="8bbebe9e-cd3b-49d7-8048-bde0a0faafa2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:58:12.483704 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:12.483660    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:14.710848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:14.710752    5071 generic.go:356] "Generic (PLEG): container finished" podID="7e3d8feb-58a2-40e0-b5b1-94acb06fe606" containerID="c79a9635fc675b303fb30614e9496e952b713e415fe8b6b259aa9c279c7f1c2e" exitCode=0
Oct 30 20:58:14.711276 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:14.710847    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz" event={"ID":"7e3d8feb-58a2-40e0-b5b1-94acb06fe606","Type":"ContainerDied","Data":"c79a9635fc675b303fb30614e9496e952b713e415fe8b6b259aa9c279c7f1c2e"}
Oct 30 20:58:14.711321 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:14.711308    5071 scope.go:122] "RemoveContainer" containerID="c79a9635fc675b303fb30614e9496e952b713e415fe8b6b259aa9c279c7f1c2e"
Oct 30 20:58:14.712810 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:14.712779    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:58:14.712937 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:14.712881    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:58:14.712937 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:14.712917    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:58:14.806813 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:14.806780    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:14.974850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:14.974788    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:15.163580 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:15.163540    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:15.668490 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:15.668451    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:15.719885 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:15.719849    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz" event={"ID":"7e3d8feb-58a2-40e0-b5b1-94acb06fe606","Type":"ContainerStarted","Data":"b83be4defe683183511b6df6616f51c17dfcffb8887f8aa3ca17748cb77a761b"}
Oct 30 20:58:15.761871 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:15.755062    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:58:15.761871 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:15.755353    5071 patch_prober.go:29] interesting pod/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 20:58:15.761871 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:15.755406    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz" podUID="7e3d8feb-58a2-40e0-b5b1-94acb06fe606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 20:58:16.750464 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:58:16.750408    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:58:16.758852 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:16.758784    5071 patch_prober.go:29] interesting pod/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 20:58:16.758987 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:16.758855    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz" podUID="7e3d8feb-58a2-40e0-b5b1-94acb06fe606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 20:58:17.755296 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:17.755259    5071 patch_prober.go:29] interesting pod/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 20:58:17.755714 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:17.755314    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz" podUID="7e3d8feb-58a2-40e0-b5b1-94acb06fe606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 20:58:17.772997 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:17.772945    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:18.755262 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:18.755218    5071 patch_prober.go:29] interesting pod/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 20:58:18.755480 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:18.755281    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz" podUID="7e3d8feb-58a2-40e0-b5b1-94acb06fe606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 20:58:19.755545 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:19.755508    5071 patch_prober.go:29] interesting pod/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 20:58:19.756199 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:19.756159    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz" podUID="7e3d8feb-58a2-40e0-b5b1-94acb06fe606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 20:58:20.554074 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:20.554026    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f6ebf5-69984c68d7-ls9bm" podUID="8bbebe9e-cd3b-49d7-8048-bde0a0faafa2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:58:20.584207 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:20.584171    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"]
Oct 30 20:58:20.697787 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:20.697717    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:20.755690 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:20.755243    5071 patch_prober.go:29] interesting pod/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 20:58:20.755690 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:20.755302    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz" podUID="7e3d8feb-58a2-40e0-b5b1-94acb06fe606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 20:58:21.759319 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:21.759288    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:58:21.759791 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:21.759772    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:58:21.760577 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:21.760554    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:58:23.198718 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:23.198685    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:27.706113 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:27.706079    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b43c84-b4788d4f4-tg275"]
Oct 30 20:58:27.739955 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:27.739920    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:27.820191 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:27.820154    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b43c84-7cc684fccf-k6nvt"]
Oct 30 20:58:27.911790 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:27.911757    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:28.096102 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:28.096061    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:28.262087 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:28.262045    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:28.708696 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:28.708657    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:30.231267 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:58:30.231235    5071 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bbebe9e_cd3b_49d7_8048_bde0a0faafa2.slice/crio-934149a7dc3af11fe6a99e9e2eba31230ffc4ef52bc894929cf10393cfe256b7.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:58:30.507751 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:30.507720    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f6ebf5-69984c68d7-ls9bm"
Oct 30 20:58:30.581170 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:30.581139    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bbebe9e-cd3b-49d7-8048-bde0a0faafa2-kube-api-access-gg45p\" (UniqueName: \"kubernetes.io/projected/8bbebe9e-cd3b-49d7-8048-bde0a0faafa2-kube-api-access-gg45p\") pod \"8bbebe9e-cd3b-49d7-8048-bde0a0faafa2\" (UID: \"8bbebe9e-cd3b-49d7-8048-bde0a0faafa2\") "
Oct 30 20:58:30.581365 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:30.581325    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bbebe9e-cd3b-49d7-8048-bde0a0faafa2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bbebe9e-cd3b-49d7-8048-bde0a0faafa2-plugins-config-volume\") pod \"8bbebe9e-cd3b-49d7-8048-bde0a0faafa2\" (UID: \"8bbebe9e-cd3b-49d7-8048-bde0a0faafa2\") "
Oct 30 20:58:30.581643 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:30.581621    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bbebe9e-cd3b-49d7-8048-bde0a0faafa2-plugins-config-volume" pod "8bbebe9e-cd3b-49d7-8048-bde0a0faafa2" (UID: "8bbebe9e-cd3b-49d7-8048-bde0a0faafa2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:58:30.581794 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:30.581780    5071 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bbebe9e-cd3b-49d7-8048-bde0a0faafa2-plugins-config-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:30.583443 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:30.583414    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bbebe9e-cd3b-49d7-8048-bde0a0faafa2-kube-api-access-gg45p" pod "8bbebe9e-cd3b-49d7-8048-bde0a0faafa2" (UID: "8bbebe9e-cd3b-49d7-8048-bde0a0faafa2"). InnerVolumeSpecName "kube-api-access-gg45p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:58:30.683097 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:30.683059    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg45p\" (UniqueName: \"kubernetes.io/projected/8bbebe9e-cd3b-49d7-8048-bde0a0faafa2-kube-api-access-gg45p\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:30.868031 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:30.867992    5071 generic.go:356] "Generic (PLEG): container finished" podID="8bbebe9e-cd3b-49d7-8048-bde0a0faafa2" containerID="934149a7dc3af11fe6a99e9e2eba31230ffc4ef52bc894929cf10393cfe256b7" exitCode=0
Oct 30 20:58:30.868213 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:30.868038    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f6ebf5-69984c68d7-ls9bm"
Oct 30 20:58:30.868213 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:30.868077    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f6ebf5-69984c68d7-ls9bm" event={"ID":"8bbebe9e-cd3b-49d7-8048-bde0a0faafa2","Type":"ContainerDied","Data":"934149a7dc3af11fe6a99e9e2eba31230ffc4ef52bc894929cf10393cfe256b7"}
Oct 30 20:58:30.868213 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:30.868133    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f6ebf5-69984c68d7-ls9bm" event={"ID":"8bbebe9e-cd3b-49d7-8048-bde0a0faafa2","Type":"ContainerDied","Data":"0fd1ae2a5c363e3a41f924caea130373d3847170ac7bed4f49a203fe5eeee660"}
Oct 30 20:58:30.868213 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:30.868157    5071 scope.go:122] "RemoveContainer" containerID="934149a7dc3af11fe6a99e9e2eba31230ffc4ef52bc894929cf10393cfe256b7"
Oct 30 20:58:30.879173 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:30.879147    5071 scope.go:122] "RemoveContainer" containerID="934149a7dc3af11fe6a99e9e2eba31230ffc4ef52bc894929cf10393cfe256b7"
Oct 30 20:58:30.879442 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:58:30.879423    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"934149a7dc3af11fe6a99e9e2eba31230ffc4ef52bc894929cf10393cfe256b7\": container with ID starting with 934149a7dc3af11fe6a99e9e2eba31230ffc4ef52bc894929cf10393cfe256b7 not found: ID does not exist" containerID="934149a7dc3af11fe6a99e9e2eba31230ffc4ef52bc894929cf10393cfe256b7"
Oct 30 20:58:30.879511 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:30.879451    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"934149a7dc3af11fe6a99e9e2eba31230ffc4ef52bc894929cf10393cfe256b7"} err="failed to get container status \"934149a7dc3af11fe6a99e9e2eba31230ffc4ef52bc894929cf10393cfe256b7\": rpc error: code = NotFound desc = could not find container \"934149a7dc3af11fe6a99e9e2eba31230ffc4ef52bc894929cf10393cfe256b7\": container with ID starting with 934149a7dc3af11fe6a99e9e2eba31230ffc4ef52bc894929cf10393cfe256b7 not found: ID does not exist"
Oct 30 20:58:31.035384 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:31.035330    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f6ebf5-69984c68d7-ls9bm"]
Oct 30 20:58:31.139941 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:31.129991    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f6ebf5-69984c68d7-ls9bm"]
Oct 30 20:58:32.202636 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:32.202602    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bbebe9e-cd3b-49d7-8048-bde0a0faafa2" path="/var/lib/kubelet/pods/8bbebe9e-cd3b-49d7-8048-bde0a0faafa2/volumes"
Oct 30 20:58:32.706714 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:32.706656    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-b43c84-b4788d4f4-tg275" podUID="78e27687-f62d-44a3-a67d-1b3397a45a00" containerName="payload-processing" containerID="cri-o://60c32c19ae9c622c638cb3375cd7801302d57f06d5733d5f6eb7cee17318635a" gracePeriod=25
Oct 30 20:58:32.821044 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:32.821006    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b43c84-7cc684fccf-k6nvt" podUID="b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4" containerName="payload-pre-processing" containerID="cri-o://294e41bfc1e3e2006f09429290fd18ce63ce1b4c6e6e264e0e61edceabc1bf0b" gracePeriod=25
Oct 30 20:58:33.512508 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:33.512465    5071 scope.go:122] "RemoveContainer" containerID="af5244846eaf79c3471770b3684e591473f95020ed88a2cf63f8e9680bc1ba2b"
Oct 30 20:58:33.556279 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:33.556189    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:33.822435 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:33.822401    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:35.093857 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:35.093803    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:35.608946 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:35.608903    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" containerID="cri-o://c5b2e09d90c044707594465e2eedcb3862b41c1fdd7797a39a4a70e9539f763b" gracePeriod=45
Oct 30 20:58:35.966090 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:35.966007    5071 generic.go:356] "Generic (PLEG): container finished" podID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerID="c5b2e09d90c044707594465e2eedcb3862b41c1fdd7797a39a4a70e9539f763b" exitCode=0
Oct 30 20:58:35.966237 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:35.966085    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" event={"ID":"5fbbb12c-4e14-45b1-a00e-84871cdcc5cc","Type":"ContainerDied","Data":"c5b2e09d90c044707594465e2eedcb3862b41c1fdd7797a39a4a70e9539f763b"}
Oct 30 20:58:36.003079 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:36.003054    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:58:36.180102 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:36.178909    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-kube-api-access-xtxks\" (UniqueName: \"kubernetes.io/projected/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-kube-api-access-xtxks\") pod \"5fbbb12c-4e14-45b1-a00e-84871cdcc5cc\" (UID: \"5fbbb12c-4e14-45b1-a00e-84871cdcc5cc\") "
Oct 30 20:58:36.180102 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:36.179025    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-model-cache\") pod \"5fbbb12c-4e14-45b1-a00e-84871cdcc5cc\" (UID: \"5fbbb12c-4e14-45b1-a00e-84871cdcc5cc\") "
Oct 30 20:58:36.180102 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:36.179118    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-tmp-dir\") pod \"5fbbb12c-4e14-45b1-a00e-84871cdcc5cc\" (UID: \"5fbbb12c-4e14-45b1-a00e-84871cdcc5cc\") "
Oct 30 20:58:36.180102 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:36.179151    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-home\" (UniqueName: \"kubernetes.io/empty-dir/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-home\") pod \"5fbbb12c-4e14-45b1-a00e-84871cdcc5cc\" (UID: \"5fbbb12c-4e14-45b1-a00e-84871cdcc5cc\") "
Oct 30 20:58:36.180102 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:36.179200    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-tls-certs\" (UniqueName: \"kubernetes.io/secret/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-tls-certs\") pod \"5fbbb12c-4e14-45b1-a00e-84871cdcc5cc\" (UID: \"5fbbb12c-4e14-45b1-a00e-84871cdcc5cc\") "
Oct 30 20:58:36.180102 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:36.179256    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-dshm\") pod \"5fbbb12c-4e14-45b1-a00e-84871cdcc5cc\" (UID: \"5fbbb12c-4e14-45b1-a00e-84871cdcc5cc\") "
Oct 30 20:58:36.181493 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:36.181463    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-tmp-dir" pod "5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" (UID: "5fbbb12c-4e14-45b1-a00e-84871cdcc5cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:58:36.181675 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:36.181491    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-home" pod "5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" (UID: "5fbbb12c-4e14-45b1-a00e-84871cdcc5cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:58:36.181758 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:36.181688    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-model-cache" pod "5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" (UID: "5fbbb12c-4e14-45b1-a00e-84871cdcc5cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:58:36.182684 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:36.182661    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-dshm" pod "5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" (UID: "5fbbb12c-4e14-45b1-a00e-84871cdcc5cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:58:36.184469 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:36.184419    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-tls-certs" pod "5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" (UID: "5fbbb12c-4e14-45b1-a00e-84871cdcc5cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:58:36.185105 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:36.185083    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-kube-api-access-xtxks" pod "5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" (UID: "5fbbb12c-4e14-45b1-a00e-84871cdcc5cc"). InnerVolumeSpecName "kube-api-access-xtxks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:58:36.281337 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:36.281304    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtxks\" (UniqueName: \"kubernetes.io/projected/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-kube-api-access-xtxks\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:36.281337 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:36.281334    5071 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-model-cache\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:36.281337 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:36.281343    5071 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-tmp-dir\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:36.281651 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:36.281352    5071 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-home\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:36.281651 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:36.281363    5071 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-tls-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:36.281651 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:36.281370    5071 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc-dshm\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:36.920242 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:36.920204    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 30 20:58:36.974579 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:36.974546    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"
Oct 30 20:58:36.974760 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:36.974580    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm" event={"ID":"5fbbb12c-4e14-45b1-a00e-84871cdcc5cc","Type":"ContainerDied","Data":"162a2c3347a858574d7ce38e7fd9936fb1a821b3a4c3326f8838a22fe208336c"}
Oct 30 20:58:36.974760 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:36.974632    5071 scope.go:122] "RemoveContainer" containerID="c5b2e09d90c044707594465e2eedcb3862b41c1fdd7797a39a4a70e9539f763b"
Oct 30 20:58:37.164426 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:37.164342    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"]
Oct 30 20:58:37.166495 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:37.166461    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:37.168043 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:37.168021    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-b43c84/distinct-w2-b43c84-kserve-7765fd4d67-pcgfm"]
Oct 30 20:58:37.388076 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:37.388041    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:37.565677 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:37.560883    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:37.915721 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:37.915681    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:38.203437 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:38.203360    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" path="/var/lib/kubelet/pods/5fbbb12c-4e14-45b1-a00e-84871cdcc5cc/volumes"
Oct 30 20:58:38.425733 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:38.425696    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd"]
Oct 30 20:58:38.426239 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:38.425937    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd" podUID="b1203632-126c-4620-8740-6f07120ad6cd" containerName="maas-api" containerID="cri-o://d1c6b4e6aa9d21e6cd5c2eb4e1e0479016f7e94d2c1a5453f58424bf038c34fa" gracePeriod=30
Oct 30 20:58:38.556662 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:38.556596    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:38.727293 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:38.726918    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b43c84-7cc684fccf-k6nvt" podUID="b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:58:38.804328 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:38.804303    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd"
Oct 30 20:58:38.962248 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:38.962211    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1203632-126c-4620-8740-6f07120ad6cd-kube-api-access-7mxxm\" (UniqueName: \"kubernetes.io/projected/b1203632-126c-4620-8740-6f07120ad6cd-kube-api-access-7mxxm\") pod \"b1203632-126c-4620-8740-6f07120ad6cd\" (UID: \"b1203632-126c-4620-8740-6f07120ad6cd\") "
Oct 30 20:58:38.962447 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:38.962351    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1203632-126c-4620-8740-6f07120ad6cd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1203632-126c-4620-8740-6f07120ad6cd-maas-api-tls\") pod \"b1203632-126c-4620-8740-6f07120ad6cd\" (UID: \"b1203632-126c-4620-8740-6f07120ad6cd\") "
Oct 30 20:58:38.962520 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:38.962489    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1203632-126c-4620-8740-6f07120ad6cd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1203632-126c-4620-8740-6f07120ad6cd-openshift-service-ca\") pod \"b1203632-126c-4620-8740-6f07120ad6cd\" (UID: \"b1203632-126c-4620-8740-6f07120ad6cd\") "
Oct 30 20:58:38.962585 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:38.962544    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1203632-126c-4620-8740-6f07120ad6cd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1203632-126c-4620-8740-6f07120ad6cd-metrics-certs\") pod \"b1203632-126c-4620-8740-6f07120ad6cd\" (UID: \"b1203632-126c-4620-8740-6f07120ad6cd\") "
Oct 30 20:58:38.964284 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:38.964252    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1203632-126c-4620-8740-6f07120ad6cd-openshift-service-ca" pod "b1203632-126c-4620-8740-6f07120ad6cd" (UID: "b1203632-126c-4620-8740-6f07120ad6cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:58:38.968373 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:38.968345    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1203632-126c-4620-8740-6f07120ad6cd-maas-api-tls" pod "b1203632-126c-4620-8740-6f07120ad6cd" (UID: "b1203632-126c-4620-8740-6f07120ad6cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:58:38.969206 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:38.969180    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1203632-126c-4620-8740-6f07120ad6cd-kube-api-access-7mxxm" pod "b1203632-126c-4620-8740-6f07120ad6cd" (UID: "b1203632-126c-4620-8740-6f07120ad6cd"). InnerVolumeSpecName "kube-api-access-7mxxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:58:38.971374 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:38.971195    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1203632-126c-4620-8740-6f07120ad6cd-metrics-certs" pod "b1203632-126c-4620-8740-6f07120ad6cd" (UID: "b1203632-126c-4620-8740-6f07120ad6cd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:58:39.002299 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:39.002259    5071 generic.go:356] "Generic (PLEG): container finished" podID="b1203632-126c-4620-8740-6f07120ad6cd" containerID="d1c6b4e6aa9d21e6cd5c2eb4e1e0479016f7e94d2c1a5453f58424bf038c34fa" exitCode=0
Oct 30 20:58:39.002474 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:39.002423    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd"
Oct 30 20:58:39.002474 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:39.002454    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd" event={"ID":"b1203632-126c-4620-8740-6f07120ad6cd","Type":"ContainerDied","Data":"d1c6b4e6aa9d21e6cd5c2eb4e1e0479016f7e94d2c1a5453f58424bf038c34fa"}
Oct 30 20:58:39.002596 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:39.002516    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd" event={"ID":"b1203632-126c-4620-8740-6f07120ad6cd","Type":"ContainerDied","Data":"fc52771cc920103370c4601c8fe2a947c66245215f72a37294054e904491f982"}
Oct 30 20:58:39.002596 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:39.002531    5071 scope.go:122] "RemoveContainer" containerID="d1c6b4e6aa9d21e6cd5c2eb4e1e0479016f7e94d2c1a5453f58424bf038c34fa"
Oct 30 20:58:39.017288 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:39.017264    5071 scope.go:122] "RemoveContainer" containerID="d1c6b4e6aa9d21e6cd5c2eb4e1e0479016f7e94d2c1a5453f58424bf038c34fa"
Oct 30 20:58:39.017637 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:58:39.017613    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1c6b4e6aa9d21e6cd5c2eb4e1e0479016f7e94d2c1a5453f58424bf038c34fa\": container with ID starting with d1c6b4e6aa9d21e6cd5c2eb4e1e0479016f7e94d2c1a5453f58424bf038c34fa not found: ID does not exist" containerID="d1c6b4e6aa9d21e6cd5c2eb4e1e0479016f7e94d2c1a5453f58424bf038c34fa"
Oct 30 20:58:39.017702 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:39.017651    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1c6b4e6aa9d21e6cd5c2eb4e1e0479016f7e94d2c1a5453f58424bf038c34fa"} err="failed to get container status \"d1c6b4e6aa9d21e6cd5c2eb4e1e0479016f7e94d2c1a5453f58424bf038c34fa\": rpc error: code = NotFound desc = could not find container \"d1c6b4e6aa9d21e6cd5c2eb4e1e0479016f7e94d2c1a5453f58424bf038c34fa\": container with ID starting with d1c6b4e6aa9d21e6cd5c2eb4e1e0479016f7e94d2c1a5453f58424bf038c34fa not found: ID does not exist"
Oct 30 20:58:39.076851 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:39.071383    5071 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1203632-126c-4620-8740-6f07120ad6cd-metrics-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:39.076851 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:39.071420    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mxxm\" (UniqueName: \"kubernetes.io/projected/b1203632-126c-4620-8740-6f07120ad6cd-kube-api-access-7mxxm\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:39.076851 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:39.071436    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1203632-126c-4620-8740-6f07120ad6cd-maas-api-tls\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:39.076851 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:39.071498    5071 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1203632-126c-4620-8740-6f07120ad6cd-openshift-service-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:39.191525 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:39.191472    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd"]
Oct 30 20:58:39.195305 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:39.195264    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-15199a-6fb8ff867d-vppgd"]
Oct 30 20:58:39.676015 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:39.675969    5071 patch_prober.go:29] interesting pod/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:58:39.676500 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:39.676041    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4" podUID="e671c902-62e8-48fc-bf30-6a2505c31167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:58:40.203857 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:40.203807    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1203632-126c-4620-8740-6f07120ad6cd" path="/var/lib/kubelet/pods/b1203632-126c-4620-8740-6f07120ad6cd/volumes"
Oct 30 20:58:40.271636 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:40.271601    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:40.527101 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:40.527009    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:40.689529 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:40.689494    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:40.699585 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:40.699549    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-b43c84-b4788d4f4-tg275" podUID="78e27687-f62d-44a3-a67d-1b3397a45a00" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:58:41.248757 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:41.248716    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:41.769065 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:41.769024    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:42.035018 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:42.034923    5071 generic.go:356] "Generic (PLEG): container finished" podID="e671c902-62e8-48fc-bf30-6a2505c31167" containerID="7154b1841877e313e01bf6c52cd7ed097f0b93e299dc64d12725273d337fdff7" exitCode=0
Oct 30 20:58:42.035018 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:42.034991    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4" event={"ID":"e671c902-62e8-48fc-bf30-6a2505c31167","Type":"ContainerDied","Data":"7154b1841877e313e01bf6c52cd7ed097f0b93e299dc64d12725273d337fdff7"}
Oct 30 20:58:42.035398 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:42.035384    5071 scope.go:122] "RemoveContainer" containerID="7154b1841877e313e01bf6c52cd7ed097f0b93e299dc64d12725273d337fdff7"
Oct 30 20:58:42.036872 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:42.036810    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:58:42.036999 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:42.036931    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:58:42.036999 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:42.036981    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:58:43.062522 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:43.062482    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4" event={"ID":"e671c902-62e8-48fc-bf30-6a2505c31167","Type":"ContainerStarted","Data":"faada5b607d3b66e8af2f5e3a6b065d5b7ad94f82af1750e6a62f84ec913408a"}
Oct 30 20:58:43.848244 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:43.848208    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 20:58:43.848624 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:43.848532    5071 patch_prober.go:29] interesting pod/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 20:58:43.848810 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:43.848785    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4" podUID="e671c902-62e8-48fc-bf30-6a2505c31167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 20:58:44.452155 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:44.451949    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:44.658849 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:44.657717    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:44.850218 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:44.850181    5071 patch_prober.go:29] interesting pod/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 20:58:44.850509 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:44.850480    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4" podUID="e671c902-62e8-48fc-bf30-6a2505c31167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 20:58:45.859012 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:45.858976    5071 patch_prober.go:29] interesting pod/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 20:58:45.859569 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:45.859073    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4" podUID="e671c902-62e8-48fc-bf30-6a2505c31167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 20:58:46.488097 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:46.488058    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:46.800181 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:58:46.800145    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:58:46.847358 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:46.847325    5071 patch_prober.go:29] interesting pod/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 20:58:46.847666 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:46.847623    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4" podUID="e671c902-62e8-48fc-bf30-6a2505c31167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 20:58:47.848015 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:47.847978    5071 patch_prober.go:29] interesting pod/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 20:58:47.848614 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:47.848582    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4" podUID="e671c902-62e8-48fc-bf30-6a2505c31167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 20:58:48.686391 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:48.686351    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b43c84-7cc684fccf-k6nvt" podUID="b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:58:48.808946 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:48.808913    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:48.873386 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:48.872953    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 20:58:48.873386 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:48.873381    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 20:58:48.874480 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:48.874453    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 20:58:48.971564 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:48.971485    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:49.111979 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:49.111944    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"]
Oct 30 20:58:49.112207 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:49.112170    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4" podUID="79131253-b9a6-41dc-b1d4-aa525c4c5e04" containerName="istio-proxy" containerID="cri-o://4c71836aa50f8860426f4b7d3f4c0d274b4e1cc86238239677e630b45fa00f24" gracePeriod=30
Oct 30 20:58:49.178307 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:49.178269    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:49.681979 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:49.681939    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:49.885504 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:49.885468    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:58:50.700115 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:50.700068    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-b43c84-b4788d4f4-tg275" podUID="78e27687-f62d-44a3-a67d-1b3397a45a00" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:58:52.432859 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:52.432784    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:53.002316 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:53.001968    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-b43c84-b4788d4f4-tg275"
Oct 30 20:58:53.081186 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:53.081154    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78e27687-f62d-44a3-a67d-1b3397a45a00-kube-api-access-8dmp6\" (UniqueName: \"kubernetes.io/projected/78e27687-f62d-44a3-a67d-1b3397a45a00-kube-api-access-8dmp6\") pod \"78e27687-f62d-44a3-a67d-1b3397a45a00\" (UID: \"78e27687-f62d-44a3-a67d-1b3397a45a00\") "
Oct 30 20:58:53.081379 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:53.081231    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78e27687-f62d-44a3-a67d-1b3397a45a00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78e27687-f62d-44a3-a67d-1b3397a45a00-plugins-config-volume\") pod \"78e27687-f62d-44a3-a67d-1b3397a45a00\" (UID: \"78e27687-f62d-44a3-a67d-1b3397a45a00\") "
Oct 30 20:58:53.081610 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:53.081581    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78e27687-f62d-44a3-a67d-1b3397a45a00-plugins-config-volume" pod "78e27687-f62d-44a3-a67d-1b3397a45a00" (UID: "78e27687-f62d-44a3-a67d-1b3397a45a00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:58:53.081958 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:53.081940    5071 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78e27687-f62d-44a3-a67d-1b3397a45a00-plugins-config-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:53.084374 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:53.084270    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78e27687-f62d-44a3-a67d-1b3397a45a00-kube-api-access-8dmp6" pod "78e27687-f62d-44a3-a67d-1b3397a45a00" (UID: "78e27687-f62d-44a3-a67d-1b3397a45a00"). InnerVolumeSpecName "kube-api-access-8dmp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:58:53.106237 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:53.106214    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b43c84-7cc684fccf-k6nvt"
Oct 30 20:58:53.152555 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:53.152500    5071 generic.go:356] "Generic (PLEG): container finished" podID="78e27687-f62d-44a3-a67d-1b3397a45a00" containerID="60c32c19ae9c622c638cb3375cd7801302d57f06d5733d5f6eb7cee17318635a" exitCode=0
Oct 30 20:58:53.152555 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:53.152549    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-b43c84-b4788d4f4-tg275"
Oct 30 20:58:53.152864 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:53.152591    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-b43c84-b4788d4f4-tg275" event={"ID":"78e27687-f62d-44a3-a67d-1b3397a45a00","Type":"ContainerDied","Data":"60c32c19ae9c622c638cb3375cd7801302d57f06d5733d5f6eb7cee17318635a"}
Oct 30 20:58:53.152864 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:53.152647    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-b43c84-b4788d4f4-tg275" event={"ID":"78e27687-f62d-44a3-a67d-1b3397a45a00","Type":"ContainerDied","Data":"ebf7c663841f67df646a83f3322760a1fc9a5d2d1549ac3fbd2f75ef6fb3e3f9"}
Oct 30 20:58:53.153342 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:53.153318    5071 scope.go:122] "RemoveContainer" containerID="60c32c19ae9c622c638cb3375cd7801302d57f06d5733d5f6eb7cee17318635a"
Oct 30 20:58:53.154458 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:53.154432    5071 generic.go:356] "Generic (PLEG): container finished" podID="b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4" containerID="294e41bfc1e3e2006f09429290fd18ce63ce1b4c6e6e264e0e61edceabc1bf0b" exitCode=0
Oct 30 20:58:53.154582 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:53.154525    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b43c84-7cc684fccf-k6nvt"
Oct 30 20:58:53.154661 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:53.154597    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b43c84-7cc684fccf-k6nvt" event={"ID":"b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4","Type":"ContainerDied","Data":"294e41bfc1e3e2006f09429290fd18ce63ce1b4c6e6e264e0e61edceabc1bf0b"}
Oct 30 20:58:53.154661 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:53.154630    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-b43c84-7cc684fccf-k6nvt" event={"ID":"b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4","Type":"ContainerDied","Data":"b5c28b9d7ed5af24cd2d9aa6bb705b4436c334202daf824cdcdbf92c7fa45832"}
Oct 30 20:58:53.165094 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:53.165074    5071 scope.go:122] "RemoveContainer" containerID="60c32c19ae9c622c638cb3375cd7801302d57f06d5733d5f6eb7cee17318635a"
Oct 30 20:58:53.165393 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:58:53.165369    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60c32c19ae9c622c638cb3375cd7801302d57f06d5733d5f6eb7cee17318635a\": container with ID starting with 60c32c19ae9c622c638cb3375cd7801302d57f06d5733d5f6eb7cee17318635a not found: ID does not exist" containerID="60c32c19ae9c622c638cb3375cd7801302d57f06d5733d5f6eb7cee17318635a"
Oct 30 20:58:53.165448 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:53.165403    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60c32c19ae9c622c638cb3375cd7801302d57f06d5733d5f6eb7cee17318635a"} err="failed to get container status \"60c32c19ae9c622c638cb3375cd7801302d57f06d5733d5f6eb7cee17318635a\": rpc error: code = NotFound desc = could not find container \"60c32c19ae9c622c638cb3375cd7801302d57f06d5733d5f6eb7cee17318635a\": container with ID starting with 60c32c19ae9c622c638cb3375cd7801302d57f06d5733d5f6eb7cee17318635a not found: ID does not exist"
Oct 30 20:58:53.165448 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:53.165430    5071 scope.go:122] "RemoveContainer" containerID="294e41bfc1e3e2006f09429290fd18ce63ce1b4c6e6e264e0e61edceabc1bf0b"
Oct 30 20:58:53.175566 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:53.175549    5071 scope.go:122] "RemoveContainer" containerID="294e41bfc1e3e2006f09429290fd18ce63ce1b4c6e6e264e0e61edceabc1bf0b"
Oct 30 20:58:53.175882 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:58:53.175846    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"294e41bfc1e3e2006f09429290fd18ce63ce1b4c6e6e264e0e61edceabc1bf0b\": container with ID starting with 294e41bfc1e3e2006f09429290fd18ce63ce1b4c6e6e264e0e61edceabc1bf0b not found: ID does not exist" containerID="294e41bfc1e3e2006f09429290fd18ce63ce1b4c6e6e264e0e61edceabc1bf0b"
Oct 30 20:58:53.175882 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:53.175873    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"294e41bfc1e3e2006f09429290fd18ce63ce1b4c6e6e264e0e61edceabc1bf0b"} err="failed to get container status \"294e41bfc1e3e2006f09429290fd18ce63ce1b4c6e6e264e0e61edceabc1bf0b\": rpc error: code = NotFound desc = could not find container \"294e41bfc1e3e2006f09429290fd18ce63ce1b4c6e6e264e0e61edceabc1bf0b\": container with ID starting with 294e41bfc1e3e2006f09429290fd18ce63ce1b4c6e6e264e0e61edceabc1bf0b not found: ID does not exist"
Oct 30 20:58:53.182346 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:53.182325    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4-kube-api-access-dz6vq\" (UniqueName: \"kubernetes.io/projected/b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4-kube-api-access-dz6vq\") pod \"b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4\" (UID: \"b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4\") "
Oct 30 20:58:53.182540 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:53.182527    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4-plugins-config-volume\") pod \"b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4\" (UID: \"b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4\") "
Oct 30 20:58:53.182864 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:53.182843    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dmp6\" (UniqueName: \"kubernetes.io/projected/78e27687-f62d-44a3-a67d-1b3397a45a00-kube-api-access-8dmp6\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:53.182981 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:53.182893    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4-plugins-config-volume" pod "b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4" (UID: "b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:58:53.184659 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:53.184637    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4-kube-api-access-dz6vq" pod "b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4" (UID: "b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4"). InnerVolumeSpecName "kube-api-access-dz6vq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:58:53.284027 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:53.283948    5071 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4-plugins-config-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:53.284027 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:53.283975    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz6vq\" (UniqueName: \"kubernetes.io/projected/b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4-kube-api-access-dz6vq\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:53.404167 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:53.404139    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b43c84-b4788d4f4-tg275"]
Oct 30 20:58:53.411423 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:53.411398    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b43c84-b4788d4f4-tg275"]
Oct 30 20:58:53.577511 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:53.577481    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b43c84-7cc684fccf-k6nvt"]
Oct 30 20:58:53.584217 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:53.584193    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-b43c84-7cc684fccf-k6nvt"]
Oct 30 20:58:54.204673 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:54.204641    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78e27687-f62d-44a3-a67d-1b3397a45a00" path="/var/lib/kubelet/pods/78e27687-f62d-44a3-a67d-1b3397a45a00/volumes"
Oct 30 20:58:54.205110 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:54.205094    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4" path="/var/lib/kubelet/pods/b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4/volumes"
Oct 30 20:58:54.412809 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:54.412784    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:58:54.457926 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:54.457893    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-3d7c11-698dd87b66-z282h"]
Oct 30 20:58:54.499634 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:54.499604    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79131253-b9a6-41dc-b1d4-aa525c4c5e04-istio-token\" (UniqueName: \"kubernetes.io/projected/79131253-b9a6-41dc-b1d4-aa525c4c5e04-istio-token\") pod \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\" (UID: \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\") "
Oct 30 20:58:54.499843 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:54.499646    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79131253-b9a6-41dc-b1d4-aa525c4c5e04-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79131253-b9a6-41dc-b1d4-aa525c4c5e04-istio-data\") pod \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\" (UID: \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\") "
Oct 30 20:58:54.499843 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:54.499725    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79131253-b9a6-41dc-b1d4-aa525c4c5e04-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79131253-b9a6-41dc-b1d4-aa525c4c5e04-istio-podinfo\") pod \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\" (UID: \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\") "
Oct 30 20:58:54.499843 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:54.499771    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79131253-b9a6-41dc-b1d4-aa525c4c5e04-kube-api-access-rd582\" (UniqueName: \"kubernetes.io/projected/79131253-b9a6-41dc-b1d4-aa525c4c5e04-kube-api-access-rd582\") pod \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\" (UID: \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\") "
Oct 30 20:58:54.499843 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:54.499802    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79131253-b9a6-41dc-b1d4-aa525c4c5e04-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79131253-b9a6-41dc-b1d4-aa525c4c5e04-workload-socket\") pod \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\" (UID: \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\") "
Oct 30 20:58:54.500089 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:54.499847    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79131253-b9a6-41dc-b1d4-aa525c4c5e04-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79131253-b9a6-41dc-b1d4-aa525c4c5e04-istiod-ca-cert\") pod \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\" (UID: \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\") "
Oct 30 20:58:54.500089 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:54.499888    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79131253-b9a6-41dc-b1d4-aa525c4c5e04-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79131253-b9a6-41dc-b1d4-aa525c4c5e04-istio-envoy\") pod \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\" (UID: \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\") "
Oct 30 20:58:54.500089 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:54.499935    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79131253-b9a6-41dc-b1d4-aa525c4c5e04-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79131253-b9a6-41dc-b1d4-aa525c4c5e04-workload-certs\") pod \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\" (UID: \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\") "
Oct 30 20:58:54.500089 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:54.500006    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79131253-b9a6-41dc-b1d4-aa525c4c5e04-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79131253-b9a6-41dc-b1d4-aa525c4c5e04-credential-socket\") pod \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\" (UID: \"79131253-b9a6-41dc-b1d4-aa525c4c5e04\") "
Oct 30 20:58:54.500089 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:54.500032    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79131253-b9a6-41dc-b1d4-aa525c4c5e04-istio-data" pod "79131253-b9a6-41dc-b1d4-aa525c4c5e04" (UID: "79131253-b9a6-41dc-b1d4-aa525c4c5e04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:58:54.500358 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:54.500140    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79131253-b9a6-41dc-b1d4-aa525c4c5e04-workload-socket" pod "79131253-b9a6-41dc-b1d4-aa525c4c5e04" (UID: "79131253-b9a6-41dc-b1d4-aa525c4c5e04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:58:54.500358 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:54.500248    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79131253-b9a6-41dc-b1d4-aa525c4c5e04-workload-certs" pod "79131253-b9a6-41dc-b1d4-aa525c4c5e04" (UID: "79131253-b9a6-41dc-b1d4-aa525c4c5e04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:58:54.500358 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:54.500296    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79131253-b9a6-41dc-b1d4-aa525c4c5e04-istiod-ca-cert" pod "79131253-b9a6-41dc-b1d4-aa525c4c5e04" (UID: "79131253-b9a6-41dc-b1d4-aa525c4c5e04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:58:54.500519 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:54.500375    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79131253-b9a6-41dc-b1d4-aa525c4c5e04-credential-socket" pod "79131253-b9a6-41dc-b1d4-aa525c4c5e04" (UID: "79131253-b9a6-41dc-b1d4-aa525c4c5e04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:58:54.500846 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:54.500717    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79131253-b9a6-41dc-b1d4-aa525c4c5e04-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:54.500846 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:54.500747    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79131253-b9a6-41dc-b1d4-aa525c4c5e04-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:54.500846 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:54.500761    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79131253-b9a6-41dc-b1d4-aa525c4c5e04-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:54.500846 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:54.500776    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79131253-b9a6-41dc-b1d4-aa525c4c5e04-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:54.500846 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:54.500790    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79131253-b9a6-41dc-b1d4-aa525c4c5e04-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:54.502521 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:54.502492    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79131253-b9a6-41dc-b1d4-aa525c4c5e04-istio-envoy" pod "79131253-b9a6-41dc-b1d4-aa525c4c5e04" (UID: "79131253-b9a6-41dc-b1d4-aa525c4c5e04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:58:54.502734 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:54.502713    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79131253-b9a6-41dc-b1d4-aa525c4c5e04-istio-podinfo" pod "79131253-b9a6-41dc-b1d4-aa525c4c5e04" (UID: "79131253-b9a6-41dc-b1d4-aa525c4c5e04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:58:54.502906 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:54.502887    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79131253-b9a6-41dc-b1d4-aa525c4c5e04-istio-token" pod "79131253-b9a6-41dc-b1d4-aa525c4c5e04" (UID: "79131253-b9a6-41dc-b1d4-aa525c4c5e04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:58:54.503076 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:54.503053    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79131253-b9a6-41dc-b1d4-aa525c4c5e04-kube-api-access-rd582" pod "79131253-b9a6-41dc-b1d4-aa525c4c5e04" (UID: "79131253-b9a6-41dc-b1d4-aa525c4c5e04"). InnerVolumeSpecName "kube-api-access-rd582". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:58:54.567406 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:54.567375    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3d7c11-5d6df9947f-xrgfn"]
Oct 30 20:58:54.602024 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:54.601991    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd582\" (UniqueName: \"kubernetes.io/projected/79131253-b9a6-41dc-b1d4-aa525c4c5e04-kube-api-access-rd582\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:54.602024 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:54.602018    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79131253-b9a6-41dc-b1d4-aa525c4c5e04-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:54.602024 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:54.602029    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79131253-b9a6-41dc-b1d4-aa525c4c5e04-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:54.602460 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:54.602038    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79131253-b9a6-41dc-b1d4-aa525c4c5e04-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:54.641952 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:54.641864    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"]
Oct 30 20:58:54.750582 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:54.750542    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:54.957904 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:54.955065    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:55.106622 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:55.106580    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:55.202538 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:55.202506    5071 generic.go:356] "Generic (PLEG): container finished" podID="79131253-b9a6-41dc-b1d4-aa525c4c5e04" containerID="4c71836aa50f8860426f4b7d3f4c0d274b4e1cc86238239677e630b45fa00f24" exitCode=0
Oct 30 20:58:55.202712 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:55.202574    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"
Oct 30 20:58:55.202712 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:55.202688    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4" event={"ID":"79131253-b9a6-41dc-b1d4-aa525c4c5e04","Type":"ContainerDied","Data":"4c71836aa50f8860426f4b7d3f4c0d274b4e1cc86238239677e630b45fa00f24"}
Oct 30 20:58:55.202871 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:55.202727    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4" event={"ID":"79131253-b9a6-41dc-b1d4-aa525c4c5e04","Type":"ContainerDied","Data":"130c83204ce27a441a8210a4f418e2c7b6ea4afedd994423fe33dd713fdfd83c"}
Oct 30 20:58:55.202871 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:55.202749    5071 scope.go:122] "RemoveContainer" containerID="4c71836aa50f8860426f4b7d3f4c0d274b4e1cc86238239677e630b45fa00f24"
Oct 30 20:58:55.223483 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:55.223457    5071 scope.go:122] "RemoveContainer" containerID="5a6213b9ad5df080a05d78effb7cc46f8bc54cec21558f29168e3cec6a358d83"
Oct 30 20:58:55.234892 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:55.234685    5071 scope.go:122] "RemoveContainer" containerID="4c71836aa50f8860426f4b7d3f4c0d274b4e1cc86238239677e630b45fa00f24"
Oct 30 20:58:55.235006 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:58:55.234985    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c71836aa50f8860426f4b7d3f4c0d274b4e1cc86238239677e630b45fa00f24\": container with ID starting with 4c71836aa50f8860426f4b7d3f4c0d274b4e1cc86238239677e630b45fa00f24 not found: ID does not exist" containerID="4c71836aa50f8860426f4b7d3f4c0d274b4e1cc86238239677e630b45fa00f24"
Oct 30 20:58:55.235048 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:55.235015    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c71836aa50f8860426f4b7d3f4c0d274b4e1cc86238239677e630b45fa00f24"} err="failed to get container status \"4c71836aa50f8860426f4b7d3f4c0d274b4e1cc86238239677e630b45fa00f24\": rpc error: code = NotFound desc = could not find container \"4c71836aa50f8860426f4b7d3f4c0d274b4e1cc86238239677e630b45fa00f24\": container with ID starting with 4c71836aa50f8860426f4b7d3f4c0d274b4e1cc86238239677e630b45fa00f24 not found: ID does not exist"
Oct 30 20:58:55.235048 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:55.235035    5071 scope.go:122] "RemoveContainer" containerID="5a6213b9ad5df080a05d78effb7cc46f8bc54cec21558f29168e3cec6a358d83"
Oct 30 20:58:55.235288 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:58:55.235268    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a6213b9ad5df080a05d78effb7cc46f8bc54cec21558f29168e3cec6a358d83\": container with ID starting with 5a6213b9ad5df080a05d78effb7cc46f8bc54cec21558f29168e3cec6a358d83 not found: ID does not exist" containerID="5a6213b9ad5df080a05d78effb7cc46f8bc54cec21558f29168e3cec6a358d83"
Oct 30 20:58:55.235345 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:55.235296    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a6213b9ad5df080a05d78effb7cc46f8bc54cec21558f29168e3cec6a358d83"} err="failed to get container status \"5a6213b9ad5df080a05d78effb7cc46f8bc54cec21558f29168e3cec6a358d83\": rpc error: code = NotFound desc = could not find container \"5a6213b9ad5df080a05d78effb7cc46f8bc54cec21558f29168e3cec6a358d83\": container with ID starting with 5a6213b9ad5df080a05d78effb7cc46f8bc54cec21558f29168e3cec6a358d83 not found: ID does not exist"
Oct 30 20:58:55.370151 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:55.370118    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"]
Oct 30 20:58:55.376027 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:55.376000    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-15199a-openshift-default-5bfdf49b4c-nzxl4"]
Oct 30 20:58:56.203152 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:56.203120    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79131253-b9a6-41dc-b1d4-aa525c4c5e04" path="/var/lib/kubelet/pods/79131253-b9a6-41dc-b1d4-aa525c4c5e04/volumes"
Oct 30 20:58:57.273734 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:57.273701    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:57.895026 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:57.894990    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:58.262316 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.262228    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7209-01131e"]
Oct 30 20:58:58.263012 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.262975    5071 cpu_manager.go:414] "containerMap: removing container" podUID="b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4" containerName="payload-pre-processing"
Oct 30 20:58:58.263012 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.263001    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4" containerName="payload-pre-processing"
Oct 30 20:58:58.263525 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.263038    5071 cpu_manager.go:414] "containerMap: removing container" podUID="b1203632-126c-4620-8740-6f07120ad6cd" containerName="maas-api"
Oct 30 20:58:58.263525 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.263046    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1203632-126c-4620-8740-6f07120ad6cd" containerName="maas-api"
Oct 30 20:58:58.263525 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.263061    5071 cpu_manager.go:414] "containerMap: removing container" podUID="79131253-b9a6-41dc-b1d4-aa525c4c5e04" containerName="istio-proxy"
Oct 30 20:58:58.263525 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.263068    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79131253-b9a6-41dc-b1d4-aa525c4c5e04" containerName="istio-proxy"
Oct 30 20:58:58.263525 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.263082    5071 cpu_manager.go:414] "containerMap: removing container" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main"
Oct 30 20:58:58.263525 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.263088    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main"
Oct 30 20:58:58.263525 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.263097    5071 cpu_manager.go:414] "containerMap: removing container" podUID="8bbebe9e-cd3b-49d7-8048-bde0a0faafa2" containerName="payload-pre-processing"
Oct 30 20:58:58.263525 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.263104    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bbebe9e-cd3b-49d7-8048-bde0a0faafa2" containerName="payload-pre-processing"
Oct 30 20:58:58.263525 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.263118    5071 cpu_manager.go:414] "containerMap: removing container" podUID="79131253-b9a6-41dc-b1d4-aa525c4c5e04" containerName="istio-proxy"
Oct 30 20:58:58.263525 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.263125    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79131253-b9a6-41dc-b1d4-aa525c4c5e04" containerName="istio-proxy"
Oct 30 20:58:58.263525 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.263144    5071 cpu_manager.go:414] "containerMap: removing container" podUID="78e27687-f62d-44a3-a67d-1b3397a45a00" containerName="payload-processing"
Oct 30 20:58:58.263525 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.263152    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78e27687-f62d-44a3-a67d-1b3397a45a00" containerName="payload-processing"
Oct 30 20:58:58.263525 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.263164    5071 cpu_manager.go:414] "containerMap: removing container" podUID="d204447d-7147-415b-abf8-853f68b51fec" containerName="istio-proxy"
Oct 30 20:58:58.263525 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.263171    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d204447d-7147-415b-abf8-853f68b51fec" containerName="istio-proxy"
Oct 30 20:58:58.263525 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.263189    5071 cpu_manager.go:414] "containerMap: removing container" podUID="a458b472-4aae-4dcc-87c9-cadd0a317061" containerName="maas-api"
Oct 30 20:58:58.263525 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.263196    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a458b472-4aae-4dcc-87c9-cadd0a317061" containerName="maas-api"
Oct 30 20:58:58.263525 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.263215    5071 cpu_manager.go:414] "containerMap: removing container" podUID="d204447d-7147-415b-abf8-853f68b51fec" containerName="istio-proxy"
Oct 30 20:58:58.263525 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.263224    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d204447d-7147-415b-abf8-853f68b51fec" containerName="istio-proxy"
Oct 30 20:58:58.263525 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.263335    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="d204447d-7147-415b-abf8-853f68b51fec" containerName="istio-proxy"
Oct 30 20:58:58.263525 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.263350    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0d1e19b-d2f0-4c2d-bcdc-9da486200fd4" containerName="payload-pre-processing"
Oct 30 20:58:58.263525 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.263358    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1203632-126c-4620-8740-6f07120ad6cd" containerName="maas-api"
Oct 30 20:58:58.263525 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.263366    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="79131253-b9a6-41dc-b1d4-aa525c4c5e04" containerName="istio-proxy"
Oct 30 20:58:58.263525 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.263372    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bbebe9e-cd3b-49d7-8048-bde0a0faafa2" containerName="payload-pre-processing"
Oct 30 20:58:58.263525 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.263378    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fbbb12c-4e14-45b1-a00e-84871cdcc5cc" containerName="main"
Oct 30 20:58:58.263525 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.263384    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="79131253-b9a6-41dc-b1d4-aa525c4c5e04" containerName="istio-proxy"
Oct 30 20:58:58.263525 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.263391    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="78e27687-f62d-44a3-a67d-1b3397a45a00" containerName="payload-processing"
Oct 30 20:58:58.263525 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.263396    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="a458b472-4aae-4dcc-87c9-cadd0a317061" containerName="maas-api"
Oct 30 20:58:58.268629 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.268611    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7209-01131e"
Oct 30 20:58:58.322283 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.322254    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7209-01131e"]
Oct 30 20:58:58.364408 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.364375    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsfnc\" (UniqueName: \"kubernetes.io/projected/5da3e261-d047-4a2d-84ec-7300b0107e38-kube-api-access-jsfnc\") pod \"test-curl-7209-01131e\" (UID: \"5da3e261-d047-4a2d-84ec-7300b0107e38\") " pod="opendatahub/test-curl-7209-01131e"
Oct 30 20:58:58.368211 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.368172    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nvzb9\""
Oct 30 20:58:58.466012 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.465981    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsfnc\" (UniqueName: \"kubernetes.io/projected/5da3e261-d047-4a2d-84ec-7300b0107e38-kube-api-access-jsfnc\") pod \"test-curl-7209-01131e\" (UID: \"5da3e261-d047-4a2d-84ec-7300b0107e38\") " pod="opendatahub/test-curl-7209-01131e"
Oct 30 20:58:58.524682 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.524605    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsfnc\" (UniqueName: \"kubernetes.io/projected/5da3e261-d047-4a2d-84ec-7300b0107e38-kube-api-access-jsfnc\") pod \"test-curl-7209-01131e\" (UID: \"5da3e261-d047-4a2d-84ec-7300b0107e38\") " pod="opendatahub/test-curl-7209-01131e"
Oct 30 20:58:58.583071 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.583032    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7209-01131e"
Oct 30 20:58:58.960228 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:58.960194    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7209-01131e"]
Oct 30 20:58:58.960712 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:58:58.960678    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5da3e261_d047_4a2d_84ec_7300b0107e38.slice/crio-0da0a409e63a9eab159586a5df42fe44a3a41c8bc3654b737bd4e860af50154e WatchSource:0}: Error finding container 0da0a409e63a9eab159586a5df42fe44a3a41c8bc3654b737bd4e860af50154e: Status 404 returned error can't find the container with id 0da0a409e63a9eab159586a5df42fe44a3a41c8bc3654b737bd4e860af50154e
Oct 30 20:58:59.252490 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:59.252268    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7209-01131e" event={"ID":"5da3e261-d047-4a2d-84ec-7300b0107e38","Type":"ContainerStarted","Data":"0da0a409e63a9eab159586a5df42fe44a3a41c8bc3654b737bd4e860af50154e"}
Oct 30 20:58:59.459113 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:59.459066    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3d7c11-698dd87b66-z282h" podUID="6af92746-24d4-4bf5-b0b4-3840ace027cd" containerName="payload-processing" containerID="cri-o://c2907229006b1f0cc980ebf57f327df53f50698fa5f54eaa57c31e2d82577d6a" gracePeriod=25
Oct 30 20:58:59.567919 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:59.567865    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3d7c11-5d6df9947f-xrgfn" podUID="ecaa3ce7-3945-4768-af31-1a0ea66825ee" containerName="payload-pre-processing" containerID="cri-o://b770efbc384b04447dd611a64201f0de3baa989ee9d967e4ef5677fe34e35f44" gracePeriod=25
Oct 30 20:58:59.679778 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:59.679615    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:59.844221 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:59.835487    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:58:59.891879 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:59.891851    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3d7c11-698dd87b66-z282h"
Oct 30 20:58:59.993298 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:59.993275    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3d7c11-5d6df9947f-xrgfn"
Oct 30 20:58:59.997529 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:59.997508    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecaa3ce7-3945-4768-af31-1a0ea66825ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecaa3ce7-3945-4768-af31-1a0ea66825ee-plugins-config-volume\") pod \"ecaa3ce7-3945-4768-af31-1a0ea66825ee\" (UID: \"ecaa3ce7-3945-4768-af31-1a0ea66825ee\") "
Oct 30 20:58:59.997671 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:59.997557    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6af92746-24d4-4bf5-b0b4-3840ace027cd-kube-api-access-l9chm\" (UniqueName: \"kubernetes.io/projected/6af92746-24d4-4bf5-b0b4-3840ace027cd-kube-api-access-l9chm\") pod \"6af92746-24d4-4bf5-b0b4-3840ace027cd\" (UID: \"6af92746-24d4-4bf5-b0b4-3840ace027cd\") "
Oct 30 20:58:59.997745 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:59.997708    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6af92746-24d4-4bf5-b0b4-3840ace027cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6af92746-24d4-4bf5-b0b4-3840ace027cd-plugins-config-volume\") pod \"6af92746-24d4-4bf5-b0b4-3840ace027cd\" (UID: \"6af92746-24d4-4bf5-b0b4-3840ace027cd\") "
Oct 30 20:58:59.997854 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:59.997834    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecaa3ce7-3945-4768-af31-1a0ea66825ee-kube-api-access-8pcnz\" (UniqueName: \"kubernetes.io/projected/ecaa3ce7-3945-4768-af31-1a0ea66825ee-kube-api-access-8pcnz\") pod \"ecaa3ce7-3945-4768-af31-1a0ea66825ee\" (UID: \"ecaa3ce7-3945-4768-af31-1a0ea66825ee\") "
Oct 30 20:58:59.998034 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:59.997921    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecaa3ce7-3945-4768-af31-1a0ea66825ee-plugins-config-volume" pod "ecaa3ce7-3945-4768-af31-1a0ea66825ee" (UID: "ecaa3ce7-3945-4768-af31-1a0ea66825ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:58:59.998127 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:59.998104    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6af92746-24d4-4bf5-b0b4-3840ace027cd-plugins-config-volume" pod "6af92746-24d4-4bf5-b0b4-3840ace027cd" (UID: "6af92746-24d4-4bf5-b0b4-3840ace027cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:58:59.998308 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:59.998290    5071 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecaa3ce7-3945-4768-af31-1a0ea66825ee-plugins-config-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:59.998386 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:58:59.998310    5071 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6af92746-24d4-4bf5-b0b4-3840ace027cd-plugins-config-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:00.000100 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:00.000069    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecaa3ce7-3945-4768-af31-1a0ea66825ee-kube-api-access-8pcnz" pod "ecaa3ce7-3945-4768-af31-1a0ea66825ee" (UID: "ecaa3ce7-3945-4768-af31-1a0ea66825ee"). InnerVolumeSpecName "kube-api-access-8pcnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:00.000191 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:00.000121    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6af92746-24d4-4bf5-b0b4-3840ace027cd-kube-api-access-l9chm" pod "6af92746-24d4-4bf5-b0b4-3840ace027cd" (UID: "6af92746-24d4-4bf5-b0b4-3840ace027cd"). InnerVolumeSpecName "kube-api-access-l9chm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:00.099183 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:00.099092    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pcnz\" (UniqueName: \"kubernetes.io/projected/ecaa3ce7-3945-4768-af31-1a0ea66825ee-kube-api-access-8pcnz\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:00.099183 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:00.099120    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9chm\" (UniqueName: \"kubernetes.io/projected/6af92746-24d4-4bf5-b0b4-3840ace027cd-kube-api-access-l9chm\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:00.261275 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:00.261235    5071 generic.go:356] "Generic (PLEG): container finished" podID="6af92746-24d4-4bf5-b0b4-3840ace027cd" containerID="c2907229006b1f0cc980ebf57f327df53f50698fa5f54eaa57c31e2d82577d6a" exitCode=0
Oct 30 20:59:00.261493 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:00.261330    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3d7c11-698dd87b66-z282h"
Oct 30 20:59:00.261493 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:00.261343    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3d7c11-698dd87b66-z282h" event={"ID":"6af92746-24d4-4bf5-b0b4-3840ace027cd","Type":"ContainerDied","Data":"c2907229006b1f0cc980ebf57f327df53f50698fa5f54eaa57c31e2d82577d6a"}
Oct 30 20:59:00.261493 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:00.261387    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3d7c11-698dd87b66-z282h" event={"ID":"6af92746-24d4-4bf5-b0b4-3840ace027cd","Type":"ContainerDied","Data":"b22117f87837785292d5a1514d3c3bde7fcef8406ada48b8d807cb4d7c5834cc"}
Oct 30 20:59:00.261493 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:00.261412    5071 scope.go:122] "RemoveContainer" containerID="c2907229006b1f0cc980ebf57f327df53f50698fa5f54eaa57c31e2d82577d6a"
Oct 30 20:59:00.263297 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:00.263270    5071 generic.go:356] "Generic (PLEG): container finished" podID="ecaa3ce7-3945-4768-af31-1a0ea66825ee" containerID="b770efbc384b04447dd611a64201f0de3baa989ee9d967e4ef5677fe34e35f44" exitCode=0
Oct 30 20:59:00.263442 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:00.263323    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3d7c11-5d6df9947f-xrgfn" event={"ID":"ecaa3ce7-3945-4768-af31-1a0ea66825ee","Type":"ContainerDied","Data":"b770efbc384b04447dd611a64201f0de3baa989ee9d967e4ef5677fe34e35f44"}
Oct 30 20:59:00.263442 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:00.263340    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3d7c11-5d6df9947f-xrgfn"
Oct 30 20:59:00.263567 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:00.263485    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3d7c11-5d6df9947f-xrgfn" event={"ID":"ecaa3ce7-3945-4768-af31-1a0ea66825ee","Type":"ContainerDied","Data":"fb3dc52d09db3994651332d4ef4139d907513bdc58143d7dc231f726392fdb1f"}
Oct 30 20:59:00.272978 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:00.272953    5071 scope.go:122] "RemoveContainer" containerID="c2907229006b1f0cc980ebf57f327df53f50698fa5f54eaa57c31e2d82577d6a"
Oct 30 20:59:00.273434 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:59:00.273410    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2907229006b1f0cc980ebf57f327df53f50698fa5f54eaa57c31e2d82577d6a\": container with ID starting with c2907229006b1f0cc980ebf57f327df53f50698fa5f54eaa57c31e2d82577d6a not found: ID does not exist" containerID="c2907229006b1f0cc980ebf57f327df53f50698fa5f54eaa57c31e2d82577d6a"
Oct 30 20:59:00.273538 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:00.273441    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2907229006b1f0cc980ebf57f327df53f50698fa5f54eaa57c31e2d82577d6a"} err="failed to get container status \"c2907229006b1f0cc980ebf57f327df53f50698fa5f54eaa57c31e2d82577d6a\": rpc error: code = NotFound desc = could not find container \"c2907229006b1f0cc980ebf57f327df53f50698fa5f54eaa57c31e2d82577d6a\": container with ID starting with c2907229006b1f0cc980ebf57f327df53f50698fa5f54eaa57c31e2d82577d6a not found: ID does not exist"
Oct 30 20:59:00.273538 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:00.273459    5071 scope.go:122] "RemoveContainer" containerID="b770efbc384b04447dd611a64201f0de3baa989ee9d967e4ef5677fe34e35f44"
Oct 30 20:59:00.282770 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:00.282753    5071 scope.go:122] "RemoveContainer" containerID="b770efbc384b04447dd611a64201f0de3baa989ee9d967e4ef5677fe34e35f44"
Oct 30 20:59:00.283082 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:59:00.283063    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b770efbc384b04447dd611a64201f0de3baa989ee9d967e4ef5677fe34e35f44\": container with ID starting with b770efbc384b04447dd611a64201f0de3baa989ee9d967e4ef5677fe34e35f44 not found: ID does not exist" containerID="b770efbc384b04447dd611a64201f0de3baa989ee9d967e4ef5677fe34e35f44"
Oct 30 20:59:00.283136 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:00.283089    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b770efbc384b04447dd611a64201f0de3baa989ee9d967e4ef5677fe34e35f44"} err="failed to get container status \"b770efbc384b04447dd611a64201f0de3baa989ee9d967e4ef5677fe34e35f44\": rpc error: code = NotFound desc = could not find container \"b770efbc384b04447dd611a64201f0de3baa989ee9d967e4ef5677fe34e35f44\": container with ID starting with b770efbc384b04447dd611a64201f0de3baa989ee9d967e4ef5677fe34e35f44 not found: ID does not exist"
Oct 30 20:59:00.429129 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:00.429037    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-3d7c11-698dd87b66-z282h"]
Oct 30 20:59:00.434856 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:00.434795    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-3d7c11-698dd87b66-z282h"]
Oct 30 20:59:00.546369 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:00.546315    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3d7c11-5d6df9947f-xrgfn"]
Oct 30 20:59:00.554438 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:00.554371    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3d7c11-5d6df9947f-xrgfn"]
Oct 30 20:59:01.783756 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:01.783726    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-386743-68d76cd7d6-xczl5"]
Oct 30 20:59:01.900552 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:01.900511    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-386743-5894c7467-txx44"]
Oct 30 20:59:02.185656 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:02.185624    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:02.276465 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:02.271545    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6af92746-24d4-4bf5-b0b4-3840ace027cd" path="/var/lib/kubelet/pods/6af92746-24d4-4bf5-b0b4-3840ace027cd/volumes"
Oct 30 20:59:02.276465 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:02.272162    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecaa3ce7-3945-4768-af31-1a0ea66825ee" path="/var/lib/kubelet/pods/ecaa3ce7-3945-4768-af31-1a0ea66825ee/volumes"
Oct 30 20:59:02.288001 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:02.287960    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7209-01131e" event={"ID":"5da3e261-d047-4a2d-84ec-7300b0107e38","Type":"ContainerStarted","Data":"3630e46ca9dd2998737bd98f393965a40f16086899fc9320402a2fdecb91fcf3"}
Oct 30 20:59:02.307455 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:02.307425    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7"]
Oct 30 20:59:02.307641 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:02.307611    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7" podUID="a31fc448-1791-4511-9b2f-c90e94928e77" containerName="maas-api" containerID="cri-o://5e249dccc783be6471b315abd64c65b9aee90c06cc103767fcee4159ba0c7f04" gracePeriod=30
Oct 30 20:59:02.348975 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:02.348923    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:02.404496 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:02.404422    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7209-01131e" podStartSLOduration=1.524272246 podStartE2EDuration="4.40440398s" totalImagesPullingTime="2.880131734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:02.401899634 +0000 UTC m=+1954.318921160" watchObservedRunningTime="2026-10-30 20:59:02.40440398 +0000 UTC m=+1954.321425521"
Oct 30 20:59:02.579779 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:02.579743    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:02.650267 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:02.650240    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7"
Oct 30 20:59:02.772276 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:02.772190    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a31fc448-1791-4511-9b2f-c90e94928e77-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a31fc448-1791-4511-9b2f-c90e94928e77-openshift-service-ca\") pod \"a31fc448-1791-4511-9b2f-c90e94928e77\" (UID: \"a31fc448-1791-4511-9b2f-c90e94928e77\") "
Oct 30 20:59:02.772427 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:02.772289    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a31fc448-1791-4511-9b2f-c90e94928e77-kube-api-access-fsfvb\" (UniqueName: \"kubernetes.io/projected/a31fc448-1791-4511-9b2f-c90e94928e77-kube-api-access-fsfvb\") pod \"a31fc448-1791-4511-9b2f-c90e94928e77\" (UID: \"a31fc448-1791-4511-9b2f-c90e94928e77\") "
Oct 30 20:59:02.772476 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:02.772446    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a31fc448-1791-4511-9b2f-c90e94928e77-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a31fc448-1791-4511-9b2f-c90e94928e77-maas-api-tls\") pod \"a31fc448-1791-4511-9b2f-c90e94928e77\" (UID: \"a31fc448-1791-4511-9b2f-c90e94928e77\") "
Oct 30 20:59:02.772536 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:02.772490    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a31fc448-1791-4511-9b2f-c90e94928e77-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a31fc448-1791-4511-9b2f-c90e94928e77-metrics-certs\") pod \"a31fc448-1791-4511-9b2f-c90e94928e77\" (UID: \"a31fc448-1791-4511-9b2f-c90e94928e77\") "
Oct 30 20:59:02.772929 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:02.772633    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a31fc448-1791-4511-9b2f-c90e94928e77-openshift-service-ca" pod "a31fc448-1791-4511-9b2f-c90e94928e77" (UID: "a31fc448-1791-4511-9b2f-c90e94928e77"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:02.773027 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:02.772949    5071 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a31fc448-1791-4511-9b2f-c90e94928e77-openshift-service-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:02.775870 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:02.775001    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a31fc448-1791-4511-9b2f-c90e94928e77-maas-api-tls" pod "a31fc448-1791-4511-9b2f-c90e94928e77" (UID: "a31fc448-1791-4511-9b2f-c90e94928e77"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:59:02.775870 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:02.775547    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a31fc448-1791-4511-9b2f-c90e94928e77-kube-api-access-fsfvb" pod "a31fc448-1791-4511-9b2f-c90e94928e77" (UID: "a31fc448-1791-4511-9b2f-c90e94928e77"). InnerVolumeSpecName "kube-api-access-fsfvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:02.776095 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:02.776071    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a31fc448-1791-4511-9b2f-c90e94928e77-metrics-certs" pod "a31fc448-1791-4511-9b2f-c90e94928e77" (UID: "a31fc448-1791-4511-9b2f-c90e94928e77"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:59:02.873874 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:02.873810    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a31fc448-1791-4511-9b2f-c90e94928e77-maas-api-tls\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:02.873874 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:02.873868    5071 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a31fc448-1791-4511-9b2f-c90e94928e77-metrics-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:02.874278 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:02.873879    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsfvb\" (UniqueName: \"kubernetes.io/projected/a31fc448-1791-4511-9b2f-c90e94928e77-kube-api-access-fsfvb\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:03.315687 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:03.315652    5071 generic.go:356] "Generic (PLEG): container finished" podID="a31fc448-1791-4511-9b2f-c90e94928e77" containerID="5e249dccc783be6471b315abd64c65b9aee90c06cc103767fcee4159ba0c7f04" exitCode=0
Oct 30 20:59:03.315951 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:03.315920    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7"
Oct 30 20:59:03.318605 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:03.318577    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7" event={"ID":"a31fc448-1791-4511-9b2f-c90e94928e77","Type":"ContainerDied","Data":"5e249dccc783be6471b315abd64c65b9aee90c06cc103767fcee4159ba0c7f04"}
Oct 30 20:59:03.319119 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:03.319094    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7" event={"ID":"a31fc448-1791-4511-9b2f-c90e94928e77","Type":"ContainerDied","Data":"9831d0c2e9e6c9864d175204ac19f87b3d3c79d98bbf7c6c54490d512a730434"}
Oct 30 20:59:03.319344 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:03.319264    5071 scope.go:122] "RemoveContainer" containerID="5e249dccc783be6471b315abd64c65b9aee90c06cc103767fcee4159ba0c7f04"
Oct 30 20:59:03.346370 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:03.346345    5071 scope.go:122] "RemoveContainer" containerID="5e249dccc783be6471b315abd64c65b9aee90c06cc103767fcee4159ba0c7f04"
Oct 30 20:59:03.347050 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:59:03.347018    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e249dccc783be6471b315abd64c65b9aee90c06cc103767fcee4159ba0c7f04\": container with ID starting with 5e249dccc783be6471b315abd64c65b9aee90c06cc103767fcee4159ba0c7f04 not found: ID does not exist" containerID="5e249dccc783be6471b315abd64c65b9aee90c06cc103767fcee4159ba0c7f04"
Oct 30 20:59:03.347186 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:03.347061    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e249dccc783be6471b315abd64c65b9aee90c06cc103767fcee4159ba0c7f04"} err="failed to get container status \"5e249dccc783be6471b315abd64c65b9aee90c06cc103767fcee4159ba0c7f04\": rpc error: code = NotFound desc = could not find container \"5e249dccc783be6471b315abd64c65b9aee90c06cc103767fcee4159ba0c7f04\": container with ID starting with 5e249dccc783be6471b315abd64c65b9aee90c06cc103767fcee4159ba0c7f04 not found: ID does not exist"
Oct 30 20:59:03.540122 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:03.540059    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7"]
Oct 30 20:59:03.550167 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:03.550121    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3d7c11-5fc4646b89-xqtf7"]
Oct 30 20:59:03.929311 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:03.929266    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7209-01131e"]
Oct 30 20:59:03.932079 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:03.932045    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7209-01131e"]
Oct 30 20:59:03.965360 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:03.965316    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:04.071188 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:04.071151    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7209-fb2fcd"]
Oct 30 20:59:04.071843 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:04.071804    5071 cpu_manager.go:414] "containerMap: removing container" podUID="ecaa3ce7-3945-4768-af31-1a0ea66825ee" containerName="payload-pre-processing"
Oct 30 20:59:04.071843 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:04.071837    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecaa3ce7-3945-4768-af31-1a0ea66825ee" containerName="payload-pre-processing"
Oct 30 20:59:04.071931 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:04.071859    5071 cpu_manager.go:414] "containerMap: removing container" podUID="6af92746-24d4-4bf5-b0b4-3840ace027cd" containerName="payload-processing"
Oct 30 20:59:04.071931 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:04.071866    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6af92746-24d4-4bf5-b0b4-3840ace027cd" containerName="payload-processing"
Oct 30 20:59:04.071931 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:04.071873    5071 cpu_manager.go:414] "containerMap: removing container" podUID="a31fc448-1791-4511-9b2f-c90e94928e77" containerName="maas-api"
Oct 30 20:59:04.071931 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:04.071877    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a31fc448-1791-4511-9b2f-c90e94928e77" containerName="maas-api"
Oct 30 20:59:04.071931 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:04.071890    5071 cpu_manager.go:414] "containerMap: removing container" podUID="5da3e261-d047-4a2d-84ec-7300b0107e38" containerName="test-curl-7209-01131e"
Oct 30 20:59:04.071931 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:04.071895    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5da3e261-d047-4a2d-84ec-7300b0107e38" containerName="test-curl-7209-01131e"
Oct 30 20:59:04.072113 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:04.072020    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="d204447d-7147-415b-abf8-853f68b51fec" containerName="istio-proxy"
Oct 30 20:59:04.072113 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:04.072038    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecaa3ce7-3945-4768-af31-1a0ea66825ee" containerName="payload-pre-processing"
Oct 30 20:59:04.072113 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:04.072048    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="6af92746-24d4-4bf5-b0b4-3840ace027cd" containerName="payload-processing"
Oct 30 20:59:04.072113 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:04.072054    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="a31fc448-1791-4511-9b2f-c90e94928e77" containerName="maas-api"
Oct 30 20:59:04.072113 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:04.072060    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="5da3e261-d047-4a2d-84ec-7300b0107e38" containerName="test-curl-7209-01131e"
Oct 30 20:59:04.076697 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:04.076680    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7209-fb2fcd"
Oct 30 20:59:04.129039 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:04.129004    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7209-fb2fcd"]
Oct 30 20:59:04.203123 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:04.203034    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a31fc448-1791-4511-9b2f-c90e94928e77" path="/var/lib/kubelet/pods/a31fc448-1791-4511-9b2f-c90e94928e77/volumes"
Oct 30 20:59:04.207033 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:04.207005    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4r9n\" (UniqueName: \"kubernetes.io/projected/d24ed347-c36e-4bbf-a9e6-66d0d9cd5ee3-kube-api-access-l4r9n\") pod \"test-curl-7209-fb2fcd\" (UID: \"d24ed347-c36e-4bbf-a9e6-66d0d9cd5ee3\") " pod="opendatahub/test-curl-7209-fb2fcd"
Oct 30 20:59:04.262542 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:04.262495    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:04.313667 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:04.313625    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4r9n\" (UniqueName: \"kubernetes.io/projected/d24ed347-c36e-4bbf-a9e6-66d0d9cd5ee3-kube-api-access-l4r9n\") pod \"test-curl-7209-fb2fcd\" (UID: \"d24ed347-c36e-4bbf-a9e6-66d0d9cd5ee3\") " pod="opendatahub/test-curl-7209-fb2fcd"
Oct 30 20:59:04.327342 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:04.327302    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7209-01131e" podUID="5da3e261-d047-4a2d-84ec-7300b0107e38" containerName="test-curl-7209-01131e" containerID="cri-o://3630e46ca9dd2998737bd98f393965a40f16086899fc9320402a2fdecb91fcf3" gracePeriod=2
Oct 30 20:59:04.372757 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:04.372725    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4r9n\" (UniqueName: \"kubernetes.io/projected/d24ed347-c36e-4bbf-a9e6-66d0d9cd5ee3-kube-api-access-l4r9n\") pod \"test-curl-7209-fb2fcd\" (UID: \"d24ed347-c36e-4bbf-a9e6-66d0d9cd5ee3\") " pod="opendatahub/test-curl-7209-fb2fcd"
Oct 30 20:59:04.389001 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:04.388976    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7209-fb2fcd"
Oct 30 20:59:04.415600 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:04.415565    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:04.607375 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:04.607336    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:04.763721 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:59:04.763691    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd24ed347_c36e_4bbf_a9e6_66d0d9cd5ee3.slice/crio-60a7fdf0b15a9eb141276973dd527531ba53254e04c00e2b3b1c31520a8cd537 WatchSource:0}: Error finding container 60a7fdf0b15a9eb141276973dd527531ba53254e04c00e2b3b1c31520a8cd537: Status 404 returned error can't find the container with id 60a7fdf0b15a9eb141276973dd527531ba53254e04c00e2b3b1c31520a8cd537
Oct 30 20:59:04.763877 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:04.763714    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7209-fb2fcd"]
Oct 30 20:59:04.775622 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:04.775571    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:04.936199 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:04.936116    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:05.335234 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:05.335202    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7209-fb2fcd" event={"ID":"d24ed347-c36e-4bbf-a9e6-66d0d9cd5ee3","Type":"ContainerStarted","Data":"e7ac241ea9a54e9363d6c28ae65bcc05ad03d1e833fc1008b9ea126e54ee2172"}
Oct 30 20:59:05.335412 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:05.335246    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7209-fb2fcd" event={"ID":"d24ed347-c36e-4bbf-a9e6-66d0d9cd5ee3","Type":"ContainerStarted","Data":"60a7fdf0b15a9eb141276973dd527531ba53254e04c00e2b3b1c31520a8cd537"}
Oct 30 20:59:05.450806 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:05.450756    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7209-fb2fcd" podStartSLOduration=1.450742781 podStartE2EDuration="1.450742781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:05.448508088 +0000 UTC m=+1957.365529629" watchObservedRunningTime="2026-10-30 20:59:05.450742781 +0000 UTC m=+1957.367764330"
Oct 30 20:59:06.305293 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:06.305253    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:06.573374 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:06.573311    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7209-01131e"
Oct 30 20:59:06.642932 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:06.642900    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5da3e261-d047-4a2d-84ec-7300b0107e38-kube-api-access-jsfnc\" (UniqueName: \"kubernetes.io/projected/5da3e261-d047-4a2d-84ec-7300b0107e38-kube-api-access-jsfnc\") pod \"5da3e261-d047-4a2d-84ec-7300b0107e38\" (UID: \"5da3e261-d047-4a2d-84ec-7300b0107e38\") "
Oct 30 20:59:06.645211 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:06.645187    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5da3e261-d047-4a2d-84ec-7300b0107e38-kube-api-access-jsfnc" pod "5da3e261-d047-4a2d-84ec-7300b0107e38" (UID: "5da3e261-d047-4a2d-84ec-7300b0107e38"). InnerVolumeSpecName "kube-api-access-jsfnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:06.745118 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:06.745087    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsfnc\" (UniqueName: \"kubernetes.io/projected/5da3e261-d047-4a2d-84ec-7300b0107e38-kube-api-access-jsfnc\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:06.784796 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:06.784752    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-386743-68d76cd7d6-xczl5" podUID="49cfe5da-09b4-4268-8938-4d6f10827d35" containerName="payload-processing" containerID="cri-o://ff757a494de40c04f6d110d862a9df9f1f16e16e49ef8749006fb984c105dae0" gracePeriod=25
Oct 30 20:59:06.900957 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:06.900855    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-386743-5894c7467-txx44" podUID="1beb9127-eb1c-41cf-8217-52ec9d5d258c" containerName="payload-pre-processing" containerID="cri-o://567e07ffad1124868e7f487828ac33a2ea753211a0936fd5098ee30337e6d1db" gracePeriod=25
Oct 30 20:59:06.980476 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:06.980433    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7209-fb2fcd"]
Oct 30 20:59:06.987287 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:06.987137    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7209-fb2fcd"]
Oct 30 20:59:07.141783 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:07.141751    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7209-a993d1"]
Oct 30 20:59:07.142378 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:07.142362    5071 cpu_manager.go:414] "containerMap: removing container" podUID="d24ed347-c36e-4bbf-a9e6-66d0d9cd5ee3" containerName="test-curl-7209-fb2fcd"
Oct 30 20:59:07.142433 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:07.142380    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d24ed347-c36e-4bbf-a9e6-66d0d9cd5ee3" containerName="test-curl-7209-fb2fcd"
Oct 30 20:59:07.142521 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:07.142510    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="d24ed347-c36e-4bbf-a9e6-66d0d9cd5ee3" containerName="test-curl-7209-fb2fcd"
Oct 30 20:59:07.145831 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:07.145798    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7209-a993d1"
Oct 30 20:59:07.200548 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:07.200516    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7209-a993d1"]
Oct 30 20:59:07.250618 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:07.250585    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scp8h\" (UniqueName: \"kubernetes.io/projected/2dc56851-26cd-42eb-a25c-c986c7f0a6a9-kube-api-access-scp8h\") pod \"test-curl-7209-a993d1\" (UID: \"2dc56851-26cd-42eb-a25c-c986c7f0a6a9\") " pod="opendatahub/test-curl-7209-a993d1"
Oct 30 20:59:07.349887 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:07.349861    5071 scope.go:122] "RemoveContainer" containerID="3630e46ca9dd2998737bd98f393965a40f16086899fc9320402a2fdecb91fcf3"
Oct 30 20:59:07.349887 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:07.349878    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7209-01131e"
Oct 30 20:59:07.350422 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:07.349855    5071 generic.go:356] "Generic (PLEG): container finished" podID="5da3e261-d047-4a2d-84ec-7300b0107e38" containerID="3630e46ca9dd2998737bd98f393965a40f16086899fc9320402a2fdecb91fcf3" exitCode=137
Oct 30 20:59:07.350422 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:07.350255    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7209-fb2fcd" podUID="d24ed347-c36e-4bbf-a9e6-66d0d9cd5ee3" containerName="test-curl-7209-fb2fcd" containerID="cri-o://e7ac241ea9a54e9363d6c28ae65bcc05ad03d1e833fc1008b9ea126e54ee2172" gracePeriod=2
Oct 30 20:59:07.351312 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:07.351290    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scp8h\" (UniqueName: \"kubernetes.io/projected/2dc56851-26cd-42eb-a25c-c986c7f0a6a9-kube-api-access-scp8h\") pod \"test-curl-7209-a993d1\" (UID: \"2dc56851-26cd-42eb-a25c-c986c7f0a6a9\") " pod="opendatahub/test-curl-7209-a993d1"
Oct 30 20:59:07.361438 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:07.361422    5071 scope.go:122] "RemoveContainer" containerID="3630e46ca9dd2998737bd98f393965a40f16086899fc9320402a2fdecb91fcf3"
Oct 30 20:59:07.361738 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:59:07.361719    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3630e46ca9dd2998737bd98f393965a40f16086899fc9320402a2fdecb91fcf3\": container with ID starting with 3630e46ca9dd2998737bd98f393965a40f16086899fc9320402a2fdecb91fcf3 not found: ID does not exist" containerID="3630e46ca9dd2998737bd98f393965a40f16086899fc9320402a2fdecb91fcf3"
Oct 30 20:59:07.361800 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:07.361746    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3630e46ca9dd2998737bd98f393965a40f16086899fc9320402a2fdecb91fcf3"} err="failed to get container status \"3630e46ca9dd2998737bd98f393965a40f16086899fc9320402a2fdecb91fcf3\": rpc error: code = NotFound desc = could not find container \"3630e46ca9dd2998737bd98f393965a40f16086899fc9320402a2fdecb91fcf3\": container with ID starting with 3630e46ca9dd2998737bd98f393965a40f16086899fc9320402a2fdecb91fcf3 not found: ID does not exist"
Oct 30 20:59:07.401010 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:59:07.400972    5071 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7209-fb2fcd\" is forbidden: User \"system:node:ip-10-0-139-149.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-149.us-west-1.compute.internal' and this object" podUID="d24ed347-c36e-4bbf-a9e6-66d0d9cd5ee3" pod="opendatahub/test-curl-7209-fb2fcd"
Oct 30 20:59:07.409390 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:07.409330    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scp8h\" (UniqueName: \"kubernetes.io/projected/2dc56851-26cd-42eb-a25c-c986c7f0a6a9-kube-api-access-scp8h\") pod \"test-curl-7209-a993d1\" (UID: \"2dc56851-26cd-42eb-a25c-c986c7f0a6a9\") " pod="opendatahub/test-curl-7209-a993d1"
Oct 30 20:59:07.451662 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:59:07.451627    5071 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7209-01131e\" is forbidden: User \"system:node:ip-10-0-139-149.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-149.us-west-1.compute.internal' and this object" podUID="5da3e261-d047-4a2d-84ec-7300b0107e38" pod="opendatahub/test-curl-7209-01131e"
Oct 30 20:59:07.459964 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:07.459941    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7209-a993d1"
Oct 30 20:59:07.502184 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:59:07.502144    5071 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7209-fb2fcd\" is forbidden: User \"system:node:ip-10-0-139-149.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-149.us-west-1.compute.internal' and this object" podUID="d24ed347-c36e-4bbf-a9e6-66d0d9cd5ee3" pod="opendatahub/test-curl-7209-fb2fcd"
Oct 30 20:59:07.838974 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:07.838943    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7209-a993d1"]
Oct 30 20:59:07.840163 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:59:07.840130    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2dc56851_26cd_42eb_a25c_c986c7f0a6a9.slice/crio-ac4a8153808b4f0b30626c4689aafd7088f91c17327595832e0dfd780fe25826 WatchSource:0}: Error finding container ac4a8153808b4f0b30626c4689aafd7088f91c17327595832e0dfd780fe25826: Status 404 returned error can't find the container with id ac4a8153808b4f0b30626c4689aafd7088f91c17327595832e0dfd780fe25826
Oct 30 20:59:08.201749 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:08.201674    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5da3e261-d047-4a2d-84ec-7300b0107e38" path="/var/lib/kubelet/pods/5da3e261-d047-4a2d-84ec-7300b0107e38/volumes"
Oct 30 20:59:08.248371 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:59:08.248339    5071 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7209-01131e\" is forbidden: User \"system:node:ip-10-0-139-149.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-149.us-west-1.compute.internal' and this object" podUID="5da3e261-d047-4a2d-84ec-7300b0107e38" pod="opendatahub/test-curl-7209-01131e"
Oct 30 20:59:08.298911 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:59:08.298874    5071 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7209-fb2fcd\" is forbidden: User \"system:node:ip-10-0-139-149.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-149.us-west-1.compute.internal' and this object" podUID="d24ed347-c36e-4bbf-a9e6-66d0d9cd5ee3" pod="opendatahub/test-curl-7209-fb2fcd"
Oct 30 20:59:08.366120 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:08.366078    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7209-a993d1" event={"ID":"2dc56851-26cd-42eb-a25c-c986c7f0a6a9","Type":"ContainerStarted","Data":"19bdb93fc9260eb3c7d63963367b4688f71f1fb0ab38e70e71f84e48e0db8a5f"}
Oct 30 20:59:08.366572 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:08.366136    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7209-a993d1" event={"ID":"2dc56851-26cd-42eb-a25c-c986c7f0a6a9","Type":"ContainerStarted","Data":"ac4a8153808b4f0b30626c4689aafd7088f91c17327595832e0dfd780fe25826"}
Oct 30 20:59:08.480875 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:08.480758    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7209-a993d1" podStartSLOduration=1.480743879 podStartE2EDuration="1.480743879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:08.478782791 +0000 UTC m=+1960.395804327" watchObservedRunningTime="2026-10-30 20:59:08.480743879 +0000 UTC m=+1960.397765437"
Oct 30 20:59:08.599803 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:08.599758    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-386743-68d76cd7d6-xczl5" podUID="49cfe5da-09b4-4268-8938-4d6f10827d35" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:59:08.599983 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:08.599899    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-386743-5894c7467-txx44" podUID="1beb9127-eb1c-41cf-8217-52ec9d5d258c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:59:09.556583 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:09.556553    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:09.645842 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:09.645805    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7209-fb2fcd"
Oct 30 20:59:09.664677 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:09.664643    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main" containerID="cri-o://e2f407d557e8c4abddacbcfcf5d08d1ef55215a27b213f82c80f7836333a9a91" gracePeriod=45
Oct 30 20:59:09.793022 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:09.792994    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d24ed347-c36e-4bbf-a9e6-66d0d9cd5ee3-kube-api-access-l4r9n\" (UniqueName: \"kubernetes.io/projected/d24ed347-c36e-4bbf-a9e6-66d0d9cd5ee3-kube-api-access-l4r9n\") pod \"d24ed347-c36e-4bbf-a9e6-66d0d9cd5ee3\" (UID: \"d24ed347-c36e-4bbf-a9e6-66d0d9cd5ee3\") "
Oct 30 20:59:09.795306 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:09.795267    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d24ed347-c36e-4bbf-a9e6-66d0d9cd5ee3-kube-api-access-l4r9n" pod "d24ed347-c36e-4bbf-a9e6-66d0d9cd5ee3" (UID: "d24ed347-c36e-4bbf-a9e6-66d0d9cd5ee3"). InnerVolumeSpecName "kube-api-access-l4r9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:09.895340 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:09.895258    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4r9n\" (UniqueName: \"kubernetes.io/projected/d24ed347-c36e-4bbf-a9e6-66d0d9cd5ee3-kube-api-access-l4r9n\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:09.923886 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:09.923799    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7209-a993d1"]
Oct 30 20:59:09.928342 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:09.928317    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7209-a993d1"]
Oct 30 20:59:09.938556 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:09.938533    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:59:10.091788 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.090634    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7209-4450a6"]
Oct 30 20:59:10.091788 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.091576    5071 cpu_manager.go:414] "containerMap: removing container" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main"
Oct 30 20:59:10.091788 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.091596    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main"
Oct 30 20:59:10.092814 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.092120    5071 cpu_manager.go:414] "containerMap: removing container" podUID="2dc56851-26cd-42eb-a25c-c986c7f0a6a9" containerName="test-curl-7209-a993d1"
Oct 30 20:59:10.092814 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.092144    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dc56851-26cd-42eb-a25c-c986c7f0a6a9" containerName="test-curl-7209-a993d1"
Oct 30 20:59:10.092814 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.092385    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerName="main"
Oct 30 20:59:10.092814 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.092412    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dc56851-26cd-42eb-a25c-c986c7f0a6a9" containerName="test-curl-7209-a993d1"
Oct 30 20:59:10.100679 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.097269    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/404848bf-abd8-421c-9419-57cb32fe6ef4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/404848bf-abd8-421c-9419-57cb32fe6ef4-tmp-dir\") pod \"404848bf-abd8-421c-9419-57cb32fe6ef4\" (UID: \"404848bf-abd8-421c-9419-57cb32fe6ef4\") "
Oct 30 20:59:10.100679 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.097469    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/404848bf-abd8-421c-9419-57cb32fe6ef4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/404848bf-abd8-421c-9419-57cb32fe6ef4-dshm\") pod \"404848bf-abd8-421c-9419-57cb32fe6ef4\" (UID: \"404848bf-abd8-421c-9419-57cb32fe6ef4\") "
Oct 30 20:59:10.100679 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.097501    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/404848bf-abd8-421c-9419-57cb32fe6ef4-kube-api-access-rvc6m\" (UniqueName: \"kubernetes.io/projected/404848bf-abd8-421c-9419-57cb32fe6ef4-kube-api-access-rvc6m\") pod \"404848bf-abd8-421c-9419-57cb32fe6ef4\" (UID: \"404848bf-abd8-421c-9419-57cb32fe6ef4\") "
Oct 30 20:59:10.100679 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.097542    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/404848bf-abd8-421c-9419-57cb32fe6ef4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/404848bf-abd8-421c-9419-57cb32fe6ef4-model-cache\") pod \"404848bf-abd8-421c-9419-57cb32fe6ef4\" (UID: \"404848bf-abd8-421c-9419-57cb32fe6ef4\") "
Oct 30 20:59:10.100679 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.097579    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/404848bf-abd8-421c-9419-57cb32fe6ef4-home\" (UniqueName: \"kubernetes.io/empty-dir/404848bf-abd8-421c-9419-57cb32fe6ef4-home\") pod \"404848bf-abd8-421c-9419-57cb32fe6ef4\" (UID: \"404848bf-abd8-421c-9419-57cb32fe6ef4\") "
Oct 30 20:59:10.100679 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.097629    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/404848bf-abd8-421c-9419-57cb32fe6ef4-tls-certs\" (UniqueName: \"kubernetes.io/secret/404848bf-abd8-421c-9419-57cb32fe6ef4-tls-certs\") pod \"404848bf-abd8-421c-9419-57cb32fe6ef4\" (UID: \"404848bf-abd8-421c-9419-57cb32fe6ef4\") "
Oct 30 20:59:10.100679 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.097621    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/404848bf-abd8-421c-9419-57cb32fe6ef4-tmp-dir" pod "404848bf-abd8-421c-9419-57cb32fe6ef4" (UID: "404848bf-abd8-421c-9419-57cb32fe6ef4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:59:10.100679 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.097863    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/404848bf-abd8-421c-9419-57cb32fe6ef4-model-cache" pod "404848bf-abd8-421c-9419-57cb32fe6ef4" (UID: "404848bf-abd8-421c-9419-57cb32fe6ef4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:59:10.100679 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.097954    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/404848bf-abd8-421c-9419-57cb32fe6ef4-home" pod "404848bf-abd8-421c-9419-57cb32fe6ef4" (UID: "404848bf-abd8-421c-9419-57cb32fe6ef4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:59:10.100679 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.098690    5071 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/404848bf-abd8-421c-9419-57cb32fe6ef4-tmp-dir\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:10.100679 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.098713    5071 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/404848bf-abd8-421c-9419-57cb32fe6ef4-model-cache\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:10.100679 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.098727    5071 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/404848bf-abd8-421c-9419-57cb32fe6ef4-home\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:10.100679 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.099666    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7209-4450a6"
Oct 30 20:59:10.117342 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.117306    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/404848bf-abd8-421c-9419-57cb32fe6ef4-kube-api-access-rvc6m" pod "404848bf-abd8-421c-9419-57cb32fe6ef4" (UID: "404848bf-abd8-421c-9419-57cb32fe6ef4"). InnerVolumeSpecName "kube-api-access-rvc6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:10.117491 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.117464    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/404848bf-abd8-421c-9419-57cb32fe6ef4-dshm" pod "404848bf-abd8-421c-9419-57cb32fe6ef4" (UID: "404848bf-abd8-421c-9419-57cb32fe6ef4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:59:10.119559 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.119535    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/404848bf-abd8-421c-9419-57cb32fe6ef4-tls-certs" pod "404848bf-abd8-421c-9419-57cb32fe6ef4" (UID: "404848bf-abd8-421c-9419-57cb32fe6ef4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:59:10.158140 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.158067    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7209-4450a6"]
Oct 30 20:59:10.205986 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.205943    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74gtd\" (UniqueName: \"kubernetes.io/projected/715ab59c-989b-4cfb-bfa7-77ca2c041e9d-kube-api-access-74gtd\") pod \"test-curl-7209-4450a6\" (UID: \"715ab59c-989b-4cfb-bfa7-77ca2c041e9d\") " pod="opendatahub/test-curl-7209-4450a6"
Oct 30 20:59:10.211607 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.211578    5071 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/404848bf-abd8-421c-9419-57cb32fe6ef4-dshm\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:10.211758 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.211611    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvc6m\" (UniqueName: \"kubernetes.io/projected/404848bf-abd8-421c-9419-57cb32fe6ef4-kube-api-access-rvc6m\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:10.211758 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.211628    5071 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/404848bf-abd8-421c-9419-57cb32fe6ef4-tls-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:10.221885 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.221837    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d24ed347-c36e-4bbf-a9e6-66d0d9cd5ee3" path="/var/lib/kubelet/pods/d24ed347-c36e-4bbf-a9e6-66d0d9cd5ee3/volumes"
Oct 30 20:59:10.313761 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.313717    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74gtd\" (UniqueName: \"kubernetes.io/projected/715ab59c-989b-4cfb-bfa7-77ca2c041e9d-kube-api-access-74gtd\") pod \"test-curl-7209-4450a6\" (UID: \"715ab59c-989b-4cfb-bfa7-77ca2c041e9d\") " pod="opendatahub/test-curl-7209-4450a6"
Oct 30 20:59:10.372408 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.372373    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74gtd\" (UniqueName: \"kubernetes.io/projected/715ab59c-989b-4cfb-bfa7-77ca2c041e9d-kube-api-access-74gtd\") pod \"test-curl-7209-4450a6\" (UID: \"715ab59c-989b-4cfb-bfa7-77ca2c041e9d\") " pod="opendatahub/test-curl-7209-4450a6"
Oct 30 20:59:10.384871 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.384833    5071 generic.go:356] "Generic (PLEG): container finished" podID="d24ed347-c36e-4bbf-a9e6-66d0d9cd5ee3" containerID="e7ac241ea9a54e9363d6c28ae65bcc05ad03d1e833fc1008b9ea126e54ee2172" exitCode=137
Oct 30 20:59:10.385020 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.384898    5071 scope.go:122] "RemoveContainer" containerID="e7ac241ea9a54e9363d6c28ae65bcc05ad03d1e833fc1008b9ea126e54ee2172"
Oct 30 20:59:10.385176 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.385151    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7209-fb2fcd"
Oct 30 20:59:10.387216 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.387194    5071 generic.go:356] "Generic (PLEG): container finished" podID="404848bf-abd8-421c-9419-57cb32fe6ef4" containerID="e2f407d557e8c4abddacbcfcf5d08d1ef55215a27b213f82c80f7836333a9a91" exitCode=0
Oct 30 20:59:10.387337 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.387240    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"
Oct 30 20:59:10.387399 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.387355    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" event={"ID":"404848bf-abd8-421c-9419-57cb32fe6ef4","Type":"ContainerDied","Data":"e2f407d557e8c4abddacbcfcf5d08d1ef55215a27b213f82c80f7836333a9a91"}
Oct 30 20:59:10.387470 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.387397    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94" event={"ID":"404848bf-abd8-421c-9419-57cb32fe6ef4","Type":"ContainerDied","Data":"379dfae7f35ec30bca15fe65272bb74b7372477068294d8a05afddf207e4d94c"}
Oct 30 20:59:10.387531 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.387458    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7209-a993d1" podUID="2dc56851-26cd-42eb-a25c-c986c7f0a6a9" containerName="test-curl-7209-a993d1" containerID="cri-o://19bdb93fc9260eb3c7d63963367b4688f71f1fb0ab38e70e71f84e48e0db8a5f" gracePeriod=2
Oct 30 20:59:10.406529 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.406506    5071 scope.go:122] "RemoveContainer" containerID="e7ac241ea9a54e9363d6c28ae65bcc05ad03d1e833fc1008b9ea126e54ee2172"
Oct 30 20:59:10.406857 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:59:10.406797    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7ac241ea9a54e9363d6c28ae65bcc05ad03d1e833fc1008b9ea126e54ee2172\": container with ID starting with e7ac241ea9a54e9363d6c28ae65bcc05ad03d1e833fc1008b9ea126e54ee2172 not found: ID does not exist" containerID="e7ac241ea9a54e9363d6c28ae65bcc05ad03d1e833fc1008b9ea126e54ee2172"
Oct 30 20:59:10.406857 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.406837    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7ac241ea9a54e9363d6c28ae65bcc05ad03d1e833fc1008b9ea126e54ee2172"} err="failed to get container status \"e7ac241ea9a54e9363d6c28ae65bcc05ad03d1e833fc1008b9ea126e54ee2172\": rpc error: code = NotFound desc = could not find container \"e7ac241ea9a54e9363d6c28ae65bcc05ad03d1e833fc1008b9ea126e54ee2172\": container with ID starting with e7ac241ea9a54e9363d6c28ae65bcc05ad03d1e833fc1008b9ea126e54ee2172 not found: ID does not exist"
Oct 30 20:59:10.406857 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.406856    5071 scope.go:122] "RemoveContainer" containerID="e2f407d557e8c4abddacbcfcf5d08d1ef55215a27b213f82c80f7836333a9a91"
Oct 30 20:59:10.416105 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.416085    5071 scope.go:122] "RemoveContainer" containerID="e2f407d557e8c4abddacbcfcf5d08d1ef55215a27b213f82c80f7836333a9a91"
Oct 30 20:59:10.416382 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:59:10.416363    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2f407d557e8c4abddacbcfcf5d08d1ef55215a27b213f82c80f7836333a9a91\": container with ID starting with e2f407d557e8c4abddacbcfcf5d08d1ef55215a27b213f82c80f7836333a9a91 not found: ID does not exist" containerID="e2f407d557e8c4abddacbcfcf5d08d1ef55215a27b213f82c80f7836333a9a91"
Oct 30 20:59:10.416468 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.416389    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2f407d557e8c4abddacbcfcf5d08d1ef55215a27b213f82c80f7836333a9a91"} err="failed to get container status \"e2f407d557e8c4abddacbcfcf5d08d1ef55215a27b213f82c80f7836333a9a91\": rpc error: code = NotFound desc = could not find container \"e2f407d557e8c4abddacbcfcf5d08d1ef55215a27b213f82c80f7836333a9a91\": container with ID starting with e2f407d557e8c4abddacbcfcf5d08d1ef55215a27b213f82c80f7836333a9a91 not found: ID does not exist"
Oct 30 20:59:10.421163 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.421145    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7209-4450a6"
Oct 30 20:59:10.556516 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.556485    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"]
Oct 30 20:59:10.561461 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.561437    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-386743/premium-w5-386743-kserve-cdc94b7fd-v9c94"]
Oct 30 20:59:10.800129 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:10.800098    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7209-4450a6"]
Oct 30 20:59:10.801398 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:59:10.801371    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod715ab59c_989b_4cfb_bfa7_77ca2c041e9d.slice/crio-981c0097ddd3556a7666941779e801136dc7c015becb85f56f15e0571b1fe39e WatchSource:0}: Error finding container 981c0097ddd3556a7666941779e801136dc7c015becb85f56f15e0571b1fe39e: Status 404 returned error can't find the container with id 981c0097ddd3556a7666941779e801136dc7c015becb85f56f15e0571b1fe39e
Oct 30 20:59:11.405969 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:11.405926    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7209-4450a6" event={"ID":"715ab59c-989b-4cfb-bfa7-77ca2c041e9d","Type":"ContainerStarted","Data":"494236c4ccabdf2d91fec464a9480124ee17e2008d9be39dd88d8602771f9d23"}
Oct 30 20:59:11.406142 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:11.405985    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7209-4450a6" event={"ID":"715ab59c-989b-4cfb-bfa7-77ca2c041e9d","Type":"ContainerStarted","Data":"981c0097ddd3556a7666941779e801136dc7c015becb85f56f15e0571b1fe39e"}
Oct 30 20:59:11.520897 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:11.520839    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7209-4450a6" podStartSLOduration=1.5208107819999999 podStartE2EDuration="1.520810782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:11.518000982 +0000 UTC m=+1963.435022519" watchObservedRunningTime="2026-10-30 20:59:11.520810782 +0000 UTC m=+1963.437832319"
Oct 30 20:59:12.202356 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:12.202310    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="404848bf-abd8-421c-9419-57cb32fe6ef4" path="/var/lib/kubelet/pods/404848bf-abd8-421c-9419-57cb32fe6ef4/volumes"
Oct 30 20:59:12.209410 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:12.209373    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:12.382518 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:12.382471    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:12.416979 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:12.416944    5071 generic.go:356] "Generic (PLEG): container finished" podID="2dc56851-26cd-42eb-a25c-c986c7f0a6a9" containerID="19bdb93fc9260eb3c7d63963367b4688f71f1fb0ab38e70e71f84e48e0db8a5f" exitCode=137
Oct 30 20:59:12.566568 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:12.564263    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:12.587891 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:12.587866    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7209-a993d1"
Oct 30 20:59:12.644402 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:12.644367    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dc56851-26cd-42eb-a25c-c986c7f0a6a9-kube-api-access-scp8h\" (UniqueName: \"kubernetes.io/projected/2dc56851-26cd-42eb-a25c-c986c7f0a6a9-kube-api-access-scp8h\") pod \"2dc56851-26cd-42eb-a25c-c986c7f0a6a9\" (UID: \"2dc56851-26cd-42eb-a25c-c986c7f0a6a9\") "
Oct 30 20:59:12.671845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:12.668724    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dc56851-26cd-42eb-a25c-c986c7f0a6a9-kube-api-access-scp8h" pod "2dc56851-26cd-42eb-a25c-c986c7f0a6a9" (UID: "2dc56851-26cd-42eb-a25c-c986c7f0a6a9"). InnerVolumeSpecName "kube-api-access-scp8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:12.745917 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:12.745879    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scp8h\" (UniqueName: \"kubernetes.io/projected/2dc56851-26cd-42eb-a25c-c986c7f0a6a9-kube-api-access-scp8h\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:13.021141 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:13.021055    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7209-4450a6"]
Oct 30 20:59:13.026877 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:13.026846    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7209-4450a6"]
Oct 30 20:59:13.424206 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:13.424177    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7209-a993d1"
Oct 30 20:59:13.424672 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:13.424240    5071 scope.go:122] "RemoveContainer" containerID="19bdb93fc9260eb3c7d63963367b4688f71f1fb0ab38e70e71f84e48e0db8a5f"
Oct 30 20:59:13.424672 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:13.424263    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7209-4450a6" podUID="715ab59c-989b-4cfb-bfa7-77ca2c041e9d" containerName="test-curl-7209-4450a6" containerID="cri-o://494236c4ccabdf2d91fec464a9480124ee17e2008d9be39dd88d8602771f9d23" gracePeriod=2
Oct 30 20:59:13.476183 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:59:13.476148    5071 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7209-4450a6\" is forbidden: User \"system:node:ip-10-0-139-149.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-149.us-west-1.compute.internal' and this object" podUID="715ab59c-989b-4cfb-bfa7-77ca2c041e9d" pod="opendatahub/test-curl-7209-4450a6"
Oct 30 20:59:13.527041 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:59:13.527006    5071 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7209-4450a6\" is forbidden: User \"system:node:ip-10-0-139-149.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-149.us-west-1.compute.internal' and this object" podUID="715ab59c-989b-4cfb-bfa7-77ca2c041e9d" pod="opendatahub/test-curl-7209-4450a6"
Oct 30 20:59:13.577813 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:59:13.577778    5071 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7209-a993d1\" is forbidden: User \"system:node:ip-10-0-139-149.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-149.us-west-1.compute.internal' and this object" podUID="2dc56851-26cd-42eb-a25c-c986c7f0a6a9" pod="opendatahub/test-curl-7209-a993d1"
Oct 30 20:59:14.201759 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:14.201721    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dc56851-26cd-42eb-a25c-c986c7f0a6a9" path="/var/lib/kubelet/pods/2dc56851-26cd-42eb-a25c-c986c7f0a6a9/volumes"
Oct 30 20:59:14.681460 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:14.681416    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:15.532798 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:15.532764    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:59:15.699760 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:15.699735    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7209-4450a6"
Oct 30 20:59:15.781685 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:15.781653    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/715ab59c-989b-4cfb-bfa7-77ca2c041e9d-kube-api-access-74gtd\" (UniqueName: \"kubernetes.io/projected/715ab59c-989b-4cfb-bfa7-77ca2c041e9d-kube-api-access-74gtd\") pod \"715ab59c-989b-4cfb-bfa7-77ca2c041e9d\" (UID: \"715ab59c-989b-4cfb-bfa7-77ca2c041e9d\") "
Oct 30 20:59:15.784377 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:15.784338    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/715ab59c-989b-4cfb-bfa7-77ca2c041e9d-kube-api-access-74gtd" pod "715ab59c-989b-4cfb-bfa7-77ca2c041e9d" (UID: "715ab59c-989b-4cfb-bfa7-77ca2c041e9d"). InnerVolumeSpecName "kube-api-access-74gtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:15.883170 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:15.883132    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74gtd\" (UniqueName: \"kubernetes.io/projected/715ab59c-989b-4cfb-bfa7-77ca2c041e9d-kube-api-access-74gtd\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:16.201391 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:16.201361    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="715ab59c-989b-4cfb-bfa7-77ca2c041e9d" path="/var/lib/kubelet/pods/715ab59c-989b-4cfb-bfa7-77ca2c041e9d/volumes"
Oct 30 20:59:16.447985 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:16.447947    5071 generic.go:356] "Generic (PLEG): container finished" podID="715ab59c-989b-4cfb-bfa7-77ca2c041e9d" containerID="494236c4ccabdf2d91fec464a9480124ee17e2008d9be39dd88d8602771f9d23" exitCode=137
Oct 30 20:59:16.448242 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:16.447991    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7209-4450a6"
Oct 30 20:59:16.448242 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:16.448045    5071 scope.go:122] "RemoveContainer" containerID="494236c4ccabdf2d91fec464a9480124ee17e2008d9be39dd88d8602771f9d23"
Oct 30 20:59:16.457885 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:16.457866    5071 scope.go:122] "RemoveContainer" containerID="494236c4ccabdf2d91fec464a9480124ee17e2008d9be39dd88d8602771f9d23"
Oct 30 20:59:16.458154 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:59:16.458135    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"494236c4ccabdf2d91fec464a9480124ee17e2008d9be39dd88d8602771f9d23\": container with ID starting with 494236c4ccabdf2d91fec464a9480124ee17e2008d9be39dd88d8602771f9d23 not found: ID does not exist" containerID="494236c4ccabdf2d91fec464a9480124ee17e2008d9be39dd88d8602771f9d23"
Oct 30 20:59:16.458232 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:16.458161    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"494236c4ccabdf2d91fec464a9480124ee17e2008d9be39dd88d8602771f9d23"} err="failed to get container status \"494236c4ccabdf2d91fec464a9480124ee17e2008d9be39dd88d8602771f9d23\": rpc error: code = NotFound desc = could not find container \"494236c4ccabdf2d91fec464a9480124ee17e2008d9be39dd88d8602771f9d23\": container with ID starting with 494236c4ccabdf2d91fec464a9480124ee17e2008d9be39dd88d8602771f9d23 not found: ID does not exist"
Oct 30 20:59:16.732434 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:59:16.732349    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:59:18.600280 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:18.600240    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-386743-5894c7467-txx44" podUID="1beb9127-eb1c-41cf-8217-52ec9d5d258c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:59:18.600770 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:18.600363    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-386743-68d76cd7d6-xczl5" podUID="49cfe5da-09b4-4268-8938-4d6f10827d35" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:59:19.970762 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:19.970723    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:22.626900 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:22.626785    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:22.795162 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:22.795121    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:24.052243 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:24.052208    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ff20fd9-fg2x8"]
Oct 30 20:59:24.101800 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:24.101771    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ff20fd9-fg2x8"]
Oct 30 20:59:24.202338 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:24.202305    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04143ede-8739-40a6-9b15-45692bc79b6a" path="/var/lib/kubelet/pods/04143ede-8739-40a6-9b15-45692bc79b6a/volumes"
Oct 30 20:59:24.397265 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:24.397180    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp"]
Oct 30 20:59:24.397427 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:24.397383    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp" podUID="b31f6255-ed4b-45af-9a92-e2de6f5ee723" containerName="maas-api" containerID="cri-o://ee31c7ba509840409ad4301374691ddbd2aa3ebe5c0b4755125130949d2bf4dd" gracePeriod=30
Oct 30 20:59:24.517053 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:24.517018    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:24.705747 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:24.705724    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp"
Oct 30 20:59:24.796367 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:24.796326    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b31f6255-ed4b-45af-9a92-e2de6f5ee723-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b31f6255-ed4b-45af-9a92-e2de6f5ee723-metrics-certs\") pod \"b31f6255-ed4b-45af-9a92-e2de6f5ee723\" (UID: \"b31f6255-ed4b-45af-9a92-e2de6f5ee723\") "
Oct 30 20:59:24.799982 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:24.799945    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b31f6255-ed4b-45af-9a92-e2de6f5ee723-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b31f6255-ed4b-45af-9a92-e2de6f5ee723-maas-api-tls\") pod \"b31f6255-ed4b-45af-9a92-e2de6f5ee723\" (UID: \"b31f6255-ed4b-45af-9a92-e2de6f5ee723\") "
Oct 30 20:59:24.808850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:24.801372    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b31f6255-ed4b-45af-9a92-e2de6f5ee723-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b31f6255-ed4b-45af-9a92-e2de6f5ee723-openshift-service-ca\") pod \"b31f6255-ed4b-45af-9a92-e2de6f5ee723\" (UID: \"b31f6255-ed4b-45af-9a92-e2de6f5ee723\") "
Oct 30 20:59:24.809155 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:24.809131    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b31f6255-ed4b-45af-9a92-e2de6f5ee723-kube-api-access-84lq6\" (UniqueName: \"kubernetes.io/projected/b31f6255-ed4b-45af-9a92-e2de6f5ee723-kube-api-access-84lq6\") pod \"b31f6255-ed4b-45af-9a92-e2de6f5ee723\" (UID: \"b31f6255-ed4b-45af-9a92-e2de6f5ee723\") "
Oct 30 20:59:24.809297 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:24.808308    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b31f6255-ed4b-45af-9a92-e2de6f5ee723-openshift-service-ca" pod "b31f6255-ed4b-45af-9a92-e2de6f5ee723" (UID: "b31f6255-ed4b-45af-9a92-e2de6f5ee723"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:24.809853 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:24.809806    5071 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b31f6255-ed4b-45af-9a92-e2de6f5ee723-openshift-service-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:24.837862 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:24.834266    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b31f6255-ed4b-45af-9a92-e2de6f5ee723-metrics-certs" pod "b31f6255-ed4b-45af-9a92-e2de6f5ee723" (UID: "b31f6255-ed4b-45af-9a92-e2de6f5ee723"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:59:24.842443 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:24.842384    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b31f6255-ed4b-45af-9a92-e2de6f5ee723-maas-api-tls" pod "b31f6255-ed4b-45af-9a92-e2de6f5ee723" (UID: "b31f6255-ed4b-45af-9a92-e2de6f5ee723"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:59:24.842597 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:24.842446    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b31f6255-ed4b-45af-9a92-e2de6f5ee723-kube-api-access-84lq6" pod "b31f6255-ed4b-45af-9a92-e2de6f5ee723" (UID: "b31f6255-ed4b-45af-9a92-e2de6f5ee723"). InnerVolumeSpecName "kube-api-access-84lq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:24.910962 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:24.910931    5071 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b31f6255-ed4b-45af-9a92-e2de6f5ee723-metrics-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:24.910962 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:24.910960    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b31f6255-ed4b-45af-9a92-e2de6f5ee723-maas-api-tls\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:24.911195 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:24.910970    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84lq6\" (UniqueName: \"kubernetes.io/projected/b31f6255-ed4b-45af-9a92-e2de6f5ee723-kube-api-access-84lq6\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:24.921067 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:24.921039    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-d9b3f6-845694f564-8jdjx"]
Oct 30 20:59:25.073889 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:25.070486    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d9b3f6-86cfc8c66c-l48wg"]
Oct 30 20:59:25.309864 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:25.309810    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:25.495103 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:25.463725    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:25.569137 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:25.569096    5071 generic.go:356] "Generic (PLEG): container finished" podID="b31f6255-ed4b-45af-9a92-e2de6f5ee723" containerID="ee31c7ba509840409ad4301374691ddbd2aa3ebe5c0b4755125130949d2bf4dd" exitCode=0
Oct 30 20:59:25.569347 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:25.569139    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp"
Oct 30 20:59:25.569347 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:25.569164    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp" event={"ID":"b31f6255-ed4b-45af-9a92-e2de6f5ee723","Type":"ContainerDied","Data":"ee31c7ba509840409ad4301374691ddbd2aa3ebe5c0b4755125130949d2bf4dd"}
Oct 30 20:59:25.569347 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:25.569225    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp" event={"ID":"b31f6255-ed4b-45af-9a92-e2de6f5ee723","Type":"ContainerDied","Data":"fa15467f81a152cd0e7d1fe162d06a13eafab64ee8c4e69152a8d153236fbc37"}
Oct 30 20:59:25.569347 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:25.569269    5071 scope.go:122] "RemoveContainer" containerID="ee31c7ba509840409ad4301374691ddbd2aa3ebe5c0b4755125130949d2bf4dd"
Oct 30 20:59:25.585769 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:25.585739    5071 scope.go:122] "RemoveContainer" containerID="ee31c7ba509840409ad4301374691ddbd2aa3ebe5c0b4755125130949d2bf4dd"
Oct 30 20:59:25.586724 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:59:25.586141    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee31c7ba509840409ad4301374691ddbd2aa3ebe5c0b4755125130949d2bf4dd\": container with ID starting with ee31c7ba509840409ad4301374691ddbd2aa3ebe5c0b4755125130949d2bf4dd not found: ID does not exist" containerID="ee31c7ba509840409ad4301374691ddbd2aa3ebe5c0b4755125130949d2bf4dd"
Oct 30 20:59:25.586724 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:25.586180    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee31c7ba509840409ad4301374691ddbd2aa3ebe5c0b4755125130949d2bf4dd"} err="failed to get container status \"ee31c7ba509840409ad4301374691ddbd2aa3ebe5c0b4755125130949d2bf4dd\": rpc error: code = NotFound desc = could not find container \"ee31c7ba509840409ad4301374691ddbd2aa3ebe5c0b4755125130949d2bf4dd\": container with ID starting with ee31c7ba509840409ad4301374691ddbd2aa3ebe5c0b4755125130949d2bf4dd not found: ID does not exist"
Oct 30 20:59:25.595662 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:59:25.595634    5071 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb31f6255_ed4b_45af_9a92_e2de6f5ee723.slice/crio-fa15467f81a152cd0e7d1fe162d06a13eafab64ee8c4e69152a8d153236fbc37\": RecentStats: unable to find data in memory cache]"
Oct 30 20:59:25.739392 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:25.739357    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp"]
Oct 30 20:59:25.744150 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:25.744113    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f6ebf5-786f7b4bfb-96hkp"]
Oct 30 20:59:26.137895 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:26.137851    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:26.202834 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:26.202803    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b31f6255-ed4b-45af-9a92-e2de6f5ee723" path="/var/lib/kubelet/pods/b31f6255-ed4b-45af-9a92-e2de6f5ee723/volumes"
Oct 30 20:59:26.376582 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:26.376542    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:26.538904 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:26.537383    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:27.080027 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:27.080003    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-386743-68d76cd7d6-xczl5"
Oct 30 20:59:27.086129 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:27.086094    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:27.153169 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:27.153133    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49cfe5da-09b4-4268-8938-4d6f10827d35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49cfe5da-09b4-4268-8938-4d6f10827d35-plugins-config-volume\") pod \"49cfe5da-09b4-4268-8938-4d6f10827d35\" (UID: \"49cfe5da-09b4-4268-8938-4d6f10827d35\") "
Oct 30 20:59:27.153667 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:27.153207    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49cfe5da-09b4-4268-8938-4d6f10827d35-kube-api-access-gpnjg\" (UniqueName: \"kubernetes.io/projected/49cfe5da-09b4-4268-8938-4d6f10827d35-kube-api-access-gpnjg\") pod \"49cfe5da-09b4-4268-8938-4d6f10827d35\" (UID: \"49cfe5da-09b4-4268-8938-4d6f10827d35\") "
Oct 30 20:59:27.153748 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:27.153707    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49cfe5da-09b4-4268-8938-4d6f10827d35-plugins-config-volume" pod "49cfe5da-09b4-4268-8938-4d6f10827d35" (UID: "49cfe5da-09b4-4268-8938-4d6f10827d35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:27.156066 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:27.156037    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49cfe5da-09b4-4268-8938-4d6f10827d35-kube-api-access-gpnjg" pod "49cfe5da-09b4-4268-8938-4d6f10827d35" (UID: "49cfe5da-09b4-4268-8938-4d6f10827d35"). InnerVolumeSpecName "kube-api-access-gpnjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:27.174886 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:27.174864    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-386743-5894c7467-txx44"
Oct 30 20:59:27.254142 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:27.254111    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1beb9127-eb1c-41cf-8217-52ec9d5d258c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1beb9127-eb1c-41cf-8217-52ec9d5d258c-plugins-config-volume\") pod \"1beb9127-eb1c-41cf-8217-52ec9d5d258c\" (UID: \"1beb9127-eb1c-41cf-8217-52ec9d5d258c\") "
Oct 30 20:59:27.254300 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:27.254288    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1beb9127-eb1c-41cf-8217-52ec9d5d258c-kube-api-access-zsj6k\" (UniqueName: \"kubernetes.io/projected/1beb9127-eb1c-41cf-8217-52ec9d5d258c-kube-api-access-zsj6k\") pod \"1beb9127-eb1c-41cf-8217-52ec9d5d258c\" (UID: \"1beb9127-eb1c-41cf-8217-52ec9d5d258c\") "
Oct 30 20:59:27.254450 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:27.254429    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1beb9127-eb1c-41cf-8217-52ec9d5d258c-plugins-config-volume" pod "1beb9127-eb1c-41cf-8217-52ec9d5d258c" (UID: "1beb9127-eb1c-41cf-8217-52ec9d5d258c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:27.254627 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:27.254613    5071 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1beb9127-eb1c-41cf-8217-52ec9d5d258c-plugins-config-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:27.254705 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:27.254631    5071 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49cfe5da-09b4-4268-8938-4d6f10827d35-plugins-config-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:27.254705 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:27.254648    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpnjg\" (UniqueName: \"kubernetes.io/projected/49cfe5da-09b4-4268-8938-4d6f10827d35-kube-api-access-gpnjg\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:27.256453 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:27.256435    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1beb9127-eb1c-41cf-8217-52ec9d5d258c-kube-api-access-zsj6k" pod "1beb9127-eb1c-41cf-8217-52ec9d5d258c" (UID: "1beb9127-eb1c-41cf-8217-52ec9d5d258c"). InnerVolumeSpecName "kube-api-access-zsj6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:27.356092 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:27.356013    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsj6k\" (UniqueName: \"kubernetes.io/projected/1beb9127-eb1c-41cf-8217-52ec9d5d258c-kube-api-access-zsj6k\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:27.446007 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:27.445974    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-9e0fb1-7f7c8cbcc6-xzvm7"]
Oct 30 20:59:27.558688 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:27.558658    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-9e0fb1-78475c7fd7-gkjhq"]
Oct 30 20:59:27.586293 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:27.586263    5071 generic.go:356] "Generic (PLEG): container finished" podID="1beb9127-eb1c-41cf-8217-52ec9d5d258c" containerID="567e07ffad1124868e7f487828ac33a2ea753211a0936fd5098ee30337e6d1db" exitCode=0
Oct 30 20:59:27.586452 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:27.586306    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-386743-5894c7467-txx44"
Oct 30 20:59:27.586452 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:27.586340    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-386743-5894c7467-txx44" event={"ID":"1beb9127-eb1c-41cf-8217-52ec9d5d258c","Type":"ContainerDied","Data":"567e07ffad1124868e7f487828ac33a2ea753211a0936fd5098ee30337e6d1db"}
Oct 30 20:59:27.586452 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:27.586389    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-386743-5894c7467-txx44" event={"ID":"1beb9127-eb1c-41cf-8217-52ec9d5d258c","Type":"ContainerDied","Data":"876d2eee5624cd47d577d5b813c23c2326957f9e60ad596c36f4d1eb0b84a4b9"}
Oct 30 20:59:27.586452 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:27.586410    5071 scope.go:122] "RemoveContainer" containerID="567e07ffad1124868e7f487828ac33a2ea753211a0936fd5098ee30337e6d1db"
Oct 30 20:59:27.587921 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:27.587898    5071 generic.go:356] "Generic (PLEG): container finished" podID="49cfe5da-09b4-4268-8938-4d6f10827d35" containerID="ff757a494de40c04f6d110d862a9df9f1f16e16e49ef8749006fb984c105dae0" exitCode=0
Oct 30 20:59:27.588039 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:27.587940    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-386743-68d76cd7d6-xczl5"
Oct 30 20:59:27.588039 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:27.588003    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-386743-68d76cd7d6-xczl5" event={"ID":"49cfe5da-09b4-4268-8938-4d6f10827d35","Type":"ContainerDied","Data":"ff757a494de40c04f6d110d862a9df9f1f16e16e49ef8749006fb984c105dae0"}
Oct 30 20:59:27.588111 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:27.588053    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-386743-68d76cd7d6-xczl5" event={"ID":"49cfe5da-09b4-4268-8938-4d6f10827d35","Type":"ContainerDied","Data":"2cb49bf7349aff98f5d26dd120a2dc4b397136492190323ea64454d3d2a7225e"}
Oct 30 20:59:27.597702 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:27.597681    5071 scope.go:122] "RemoveContainer" containerID="567e07ffad1124868e7f487828ac33a2ea753211a0936fd5098ee30337e6d1db"
Oct 30 20:59:27.599200 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:59:27.599177    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"567e07ffad1124868e7f487828ac33a2ea753211a0936fd5098ee30337e6d1db\": container with ID starting with 567e07ffad1124868e7f487828ac33a2ea753211a0936fd5098ee30337e6d1db not found: ID does not exist" containerID="567e07ffad1124868e7f487828ac33a2ea753211a0936fd5098ee30337e6d1db"
Oct 30 20:59:27.599313 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:27.599210    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"567e07ffad1124868e7f487828ac33a2ea753211a0936fd5098ee30337e6d1db"} err="failed to get container status \"567e07ffad1124868e7f487828ac33a2ea753211a0936fd5098ee30337e6d1db\": rpc error: code = NotFound desc = could not find container \"567e07ffad1124868e7f487828ac33a2ea753211a0936fd5098ee30337e6d1db\": container with ID starting with 567e07ffad1124868e7f487828ac33a2ea753211a0936fd5098ee30337e6d1db not found: ID does not exist"
Oct 30 20:59:27.599313 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:27.599251    5071 scope.go:122] "RemoveContainer" containerID="ff757a494de40c04f6d110d862a9df9f1f16e16e49ef8749006fb984c105dae0"
Oct 30 20:59:27.611766 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:27.611746    5071 scope.go:122] "RemoveContainer" containerID="ff757a494de40c04f6d110d862a9df9f1f16e16e49ef8749006fb984c105dae0"
Oct 30 20:59:27.612083 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:59:27.612058    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff757a494de40c04f6d110d862a9df9f1f16e16e49ef8749006fb984c105dae0\": container with ID starting with ff757a494de40c04f6d110d862a9df9f1f16e16e49ef8749006fb984c105dae0 not found: ID does not exist" containerID="ff757a494de40c04f6d110d862a9df9f1f16e16e49ef8749006fb984c105dae0"
Oct 30 20:59:27.612164 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:27.612094    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff757a494de40c04f6d110d862a9df9f1f16e16e49ef8749006fb984c105dae0"} err="failed to get container status \"ff757a494de40c04f6d110d862a9df9f1f16e16e49ef8749006fb984c105dae0\": rpc error: code = NotFound desc = could not find container \"ff757a494de40c04f6d110d862a9df9f1f16e16e49ef8749006fb984c105dae0\": container with ID starting with ff757a494de40c04f6d110d862a9df9f1f16e16e49ef8749006fb984c105dae0 not found: ID does not exist"
Oct 30 20:59:27.793192 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:27.793161    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-386743-68d76cd7d6-xczl5"]
Oct 30 20:59:27.798796 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:27.798766    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-386743-68d76cd7d6-xczl5"]
Oct 30 20:59:27.858380 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:27.858343    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:27.917674 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:27.917570    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-386743-5894c7467-txx44"]
Oct 30 20:59:27.923021 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:27.922915    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-386743-5894c7467-txx44"]
Oct 30 20:59:28.052882 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:28.040272    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:28.204021 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:28.203951    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1beb9127-eb1c-41cf-8217-52ec9d5d258c" path="/var/lib/kubelet/pods/1beb9127-eb1c-41cf-8217-52ec9d5d258c/volumes"
Oct 30 20:59:28.204398 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:28.204356    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49cfe5da-09b4-4268-8938-4d6f10827d35" path="/var/lib/kubelet/pods/49cfe5da-09b4-4268-8938-4d6f10827d35/volumes"
Oct 30 20:59:29.921626 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:29.921589    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d9b3f6-845694f564-8jdjx" podUID="902523f4-4967-4ef2-97be-632d3a0c6383" containerName="payload-processing" containerID="cri-o://8f0bf82524098ef28f1ccc1e285f05d2c56e00c2eaae032e2e5af1163ac6f331" gracePeriod=25
Oct 30 20:59:30.071247 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:30.071192    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d9b3f6-86cfc8c66c-l48wg" podUID="940a58d0-5deb-4bc9-8832-5a17fa300539" containerName="payload-pre-processing" containerID="cri-o://a3dab290335c6e2959d6be89508aa8433cd6ee89977f3d0b52986fb0abd5c7d8" gracePeriod=25
Oct 30 20:59:30.215158 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:30.215134    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d9b3f6-845694f564-8jdjx"
Oct 30 20:59:30.306806 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:30.306774    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/902523f4-4967-4ef2-97be-632d3a0c6383-kube-api-access-txj2z\" (UniqueName: \"kubernetes.io/projected/902523f4-4967-4ef2-97be-632d3a0c6383-kube-api-access-txj2z\") pod \"902523f4-4967-4ef2-97be-632d3a0c6383\" (UID: \"902523f4-4967-4ef2-97be-632d3a0c6383\") "
Oct 30 20:59:30.307089 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:30.307071    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/902523f4-4967-4ef2-97be-632d3a0c6383-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/902523f4-4967-4ef2-97be-632d3a0c6383-plugins-config-volume\") pod \"902523f4-4967-4ef2-97be-632d3a0c6383\" (UID: \"902523f4-4967-4ef2-97be-632d3a0c6383\") "
Oct 30 20:59:30.307579 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:30.307552    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/902523f4-4967-4ef2-97be-632d3a0c6383-plugins-config-volume" pod "902523f4-4967-4ef2-97be-632d3a0c6383" (UID: "902523f4-4967-4ef2-97be-632d3a0c6383"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:30.308215 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:30.308187    5071 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/902523f4-4967-4ef2-97be-632d3a0c6383-plugins-config-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:30.309309 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:30.309288    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/902523f4-4967-4ef2-97be-632d3a0c6383-kube-api-access-txj2z" pod "902523f4-4967-4ef2-97be-632d3a0c6383" (UID: "902523f4-4967-4ef2-97be-632d3a0c6383"). InnerVolumeSpecName "kube-api-access-txj2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:30.331185 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:30.330944    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d9b3f6-86cfc8c66c-l48wg"
Oct 30 20:59:30.409093 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:30.409063    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/940a58d0-5deb-4bc9-8832-5a17fa300539-kube-api-access-sn59p\" (UniqueName: \"kubernetes.io/projected/940a58d0-5deb-4bc9-8832-5a17fa300539-kube-api-access-sn59p\") pod \"940a58d0-5deb-4bc9-8832-5a17fa300539\" (UID: \"940a58d0-5deb-4bc9-8832-5a17fa300539\") "
Oct 30 20:59:30.409260 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:30.409226    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/940a58d0-5deb-4bc9-8832-5a17fa300539-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/940a58d0-5deb-4bc9-8832-5a17fa300539-plugins-config-volume\") pod \"940a58d0-5deb-4bc9-8832-5a17fa300539\" (UID: \"940a58d0-5deb-4bc9-8832-5a17fa300539\") "
Oct 30 20:59:30.409581 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:30.409552    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/940a58d0-5deb-4bc9-8832-5a17fa300539-plugins-config-volume" pod "940a58d0-5deb-4bc9-8832-5a17fa300539" (UID: "940a58d0-5deb-4bc9-8832-5a17fa300539"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:30.409726 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:30.409710    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txj2z\" (UniqueName: \"kubernetes.io/projected/902523f4-4967-4ef2-97be-632d3a0c6383-kube-api-access-txj2z\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:30.409778 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:30.409734    5071 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/940a58d0-5deb-4bc9-8832-5a17fa300539-plugins-config-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:30.411381 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:30.411362    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/940a58d0-5deb-4bc9-8832-5a17fa300539-kube-api-access-sn59p" pod "940a58d0-5deb-4bc9-8832-5a17fa300539" (UID: "940a58d0-5deb-4bc9-8832-5a17fa300539"). InnerVolumeSpecName "kube-api-access-sn59p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:30.510525 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:30.510451    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn59p\" (UniqueName: \"kubernetes.io/projected/940a58d0-5deb-4bc9-8832-5a17fa300539-kube-api-access-sn59p\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:30.613570 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:30.613538    5071 generic.go:356] "Generic (PLEG): container finished" podID="940a58d0-5deb-4bc9-8832-5a17fa300539" containerID="a3dab290335c6e2959d6be89508aa8433cd6ee89977f3d0b52986fb0abd5c7d8" exitCode=0
Oct 30 20:59:30.613769 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:30.613590    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d9b3f6-86cfc8c66c-l48wg"
Oct 30 20:59:30.613769 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:30.613609    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d9b3f6-86cfc8c66c-l48wg" event={"ID":"940a58d0-5deb-4bc9-8832-5a17fa300539","Type":"ContainerDied","Data":"a3dab290335c6e2959d6be89508aa8433cd6ee89977f3d0b52986fb0abd5c7d8"}
Oct 30 20:59:30.613769 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:30.613657    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d9b3f6-86cfc8c66c-l48wg" event={"ID":"940a58d0-5deb-4bc9-8832-5a17fa300539","Type":"ContainerDied","Data":"582a789d3aea6cc26912c29e8bc1353c2853cdd7e1084cc650fbcff87a13c5d0"}
Oct 30 20:59:30.613769 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:30.613679    5071 scope.go:122] "RemoveContainer" containerID="a3dab290335c6e2959d6be89508aa8433cd6ee89977f3d0b52986fb0abd5c7d8"
Oct 30 20:59:30.615155 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:30.615131    5071 generic.go:356] "Generic (PLEG): container finished" podID="902523f4-4967-4ef2-97be-632d3a0c6383" containerID="8f0bf82524098ef28f1ccc1e285f05d2c56e00c2eaae032e2e5af1163ac6f331" exitCode=0
Oct 30 20:59:30.615265 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:30.615175    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d9b3f6-845694f564-8jdjx"
Oct 30 20:59:30.615265 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:30.615198    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d9b3f6-845694f564-8jdjx" event={"ID":"902523f4-4967-4ef2-97be-632d3a0c6383","Type":"ContainerDied","Data":"8f0bf82524098ef28f1ccc1e285f05d2c56e00c2eaae032e2e5af1163ac6f331"}
Oct 30 20:59:30.615265 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:30.615244    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d9b3f6-845694f564-8jdjx" event={"ID":"902523f4-4967-4ef2-97be-632d3a0c6383","Type":"ContainerDied","Data":"f3498ceb33be705edde86b807f7bb73afd8df3bd7b090206fb40fa0197731f64"}
Oct 30 20:59:30.623983 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:30.623959    5071 scope.go:122] "RemoveContainer" containerID="a3dab290335c6e2959d6be89508aa8433cd6ee89977f3d0b52986fb0abd5c7d8"
Oct 30 20:59:30.624279 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:59:30.624260    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3dab290335c6e2959d6be89508aa8433cd6ee89977f3d0b52986fb0abd5c7d8\": container with ID starting with a3dab290335c6e2959d6be89508aa8433cd6ee89977f3d0b52986fb0abd5c7d8 not found: ID does not exist" containerID="a3dab290335c6e2959d6be89508aa8433cd6ee89977f3d0b52986fb0abd5c7d8"
Oct 30 20:59:30.624365 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:30.624290    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3dab290335c6e2959d6be89508aa8433cd6ee89977f3d0b52986fb0abd5c7d8"} err="failed to get container status \"a3dab290335c6e2959d6be89508aa8433cd6ee89977f3d0b52986fb0abd5c7d8\": rpc error: code = NotFound desc = could not find container \"a3dab290335c6e2959d6be89508aa8433cd6ee89977f3d0b52986fb0abd5c7d8\": container with ID starting with a3dab290335c6e2959d6be89508aa8433cd6ee89977f3d0b52986fb0abd5c7d8 not found: ID does not exist"
Oct 30 20:59:30.624365 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:30.624307    5071 scope.go:122] "RemoveContainer" containerID="8f0bf82524098ef28f1ccc1e285f05d2c56e00c2eaae032e2e5af1163ac6f331"
Oct 30 20:59:30.636735 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:30.636713    5071 scope.go:122] "RemoveContainer" containerID="8f0bf82524098ef28f1ccc1e285f05d2c56e00c2eaae032e2e5af1163ac6f331"
Oct 30 20:59:30.637113 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:59:30.637086    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f0bf82524098ef28f1ccc1e285f05d2c56e00c2eaae032e2e5af1163ac6f331\": container with ID starting with 8f0bf82524098ef28f1ccc1e285f05d2c56e00c2eaae032e2e5af1163ac6f331 not found: ID does not exist" containerID="8f0bf82524098ef28f1ccc1e285f05d2c56e00c2eaae032e2e5af1163ac6f331"
Oct 30 20:59:30.637183 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:30.637129    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f0bf82524098ef28f1ccc1e285f05d2c56e00c2eaae032e2e5af1163ac6f331"} err="failed to get container status \"8f0bf82524098ef28f1ccc1e285f05d2c56e00c2eaae032e2e5af1163ac6f331\": rpc error: code = NotFound desc = could not find container \"8f0bf82524098ef28f1ccc1e285f05d2c56e00c2eaae032e2e5af1163ac6f331\": container with ID starting with 8f0bf82524098ef28f1ccc1e285f05d2c56e00c2eaae032e2e5af1163ac6f331 not found: ID does not exist"
Oct 30 20:59:30.781054 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:30.780968    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d9b3f6-86cfc8c66c-l48wg"]
Oct 30 20:59:30.787192 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:30.787159    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d9b3f6-86cfc8c66c-l48wg"]
Oct 30 20:59:30.897576 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:30.897541    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-d9b3f6-845694f564-8jdjx"]
Oct 30 20:59:30.900109 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:30.900084    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-d9b3f6-845694f564-8jdjx"]
Oct 30 20:59:31.008694 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:31.008662    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:31.752653 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:31.752614    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:31.917247 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:31.917211    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:32.202168 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:32.202124    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="902523f4-4967-4ef2-97be-632d3a0c6383" path="/var/lib/kubelet/pods/902523f4-4967-4ef2-97be-632d3a0c6383/volumes"
Oct 30 20:59:32.202738 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:32.202717    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="940a58d0-5deb-4bc9-8832-5a17fa300539" path="/var/lib/kubelet/pods/940a58d0-5deb-4bc9-8832-5a17fa300539/volumes"
Oct 30 20:59:32.291483 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:32.291430    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:32.447111 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:32.447065    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-9e0fb1-7f7c8cbcc6-xzvm7" podUID="d6cafaf5-279b-4ec5-b830-b27a7c00eb19" containerName="payload-processing" containerID="cri-o://bdcd52dbb88e058054665f444178954745819b593e30fb45fcdc813685bd7346" gracePeriod=25
Oct 30 20:59:32.559930 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:32.559885    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-9e0fb1-78475c7fd7-gkjhq" podUID="dc24a7fe-b98b-4e31-ac64-3b80da898040" containerName="payload-pre-processing" containerID="cri-o://1b6642ae87b59e8a97266a4e379238b11b4563a49b0f5dde44127b399129a45b" gracePeriod=25
Oct 30 20:59:32.607357 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:32.607320    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:32.636069 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:32.636033    5071 generic.go:356] "Generic (PLEG): container finished" podID="d6cafaf5-279b-4ec5-b830-b27a7c00eb19" containerID="bdcd52dbb88e058054665f444178954745819b593e30fb45fcdc813685bd7346" exitCode=0
Oct 30 20:59:32.636270 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:32.636251    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-9e0fb1-7f7c8cbcc6-xzvm7" event={"ID":"d6cafaf5-279b-4ec5-b830-b27a7c00eb19","Type":"ContainerDied","Data":"bdcd52dbb88e058054665f444178954745819b593e30fb45fcdc813685bd7346"}
Oct 30 20:59:32.717369 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:32.717307    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-9e0fb1-7f7c8cbcc6-xzvm7"
Oct 30 20:59:32.769856 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:32.766229    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6cafaf5-279b-4ec5-b830-b27a7c00eb19-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6cafaf5-279b-4ec5-b830-b27a7c00eb19-plugins-config-volume\") pod \"d6cafaf5-279b-4ec5-b830-b27a7c00eb19\" (UID: \"d6cafaf5-279b-4ec5-b830-b27a7c00eb19\") "
Oct 30 20:59:32.769856 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:32.766421    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6cafaf5-279b-4ec5-b830-b27a7c00eb19-kube-api-access-4qf9b\" (UniqueName: \"kubernetes.io/projected/d6cafaf5-279b-4ec5-b830-b27a7c00eb19-kube-api-access-4qf9b\") pod \"d6cafaf5-279b-4ec5-b830-b27a7c00eb19\" (UID: \"d6cafaf5-279b-4ec5-b830-b27a7c00eb19\") "
Oct 30 20:59:32.786880 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:32.774106    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6cafaf5-279b-4ec5-b830-b27a7c00eb19-plugins-config-volume" pod "d6cafaf5-279b-4ec5-b830-b27a7c00eb19" (UID: "d6cafaf5-279b-4ec5-b830-b27a7c00eb19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:32.844925 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:32.831280    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6cafaf5-279b-4ec5-b830-b27a7c00eb19-kube-api-access-4qf9b" pod "d6cafaf5-279b-4ec5-b830-b27a7c00eb19" (UID: "d6cafaf5-279b-4ec5-b830-b27a7c00eb19"). InnerVolumeSpecName "kube-api-access-4qf9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:32.867970 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:32.867931    5071 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6cafaf5-279b-4ec5-b830-b27a7c00eb19-plugins-config-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:32.867970 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:32.867964    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qf9b\" (UniqueName: \"kubernetes.io/projected/d6cafaf5-279b-4ec5-b830-b27a7c00eb19-kube-api-access-4qf9b\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:32.912144 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:32.912112    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-9e0fb1-78475c7fd7-gkjhq"
Oct 30 20:59:32.959848 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:32.944294    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:32.972955 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:32.970127    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc24a7fe-b98b-4e31-ac64-3b80da898040-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc24a7fe-b98b-4e31-ac64-3b80da898040-plugins-config-volume\") pod \"dc24a7fe-b98b-4e31-ac64-3b80da898040\" (UID: \"dc24a7fe-b98b-4e31-ac64-3b80da898040\") "
Oct 30 20:59:32.972955 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:32.970222    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc24a7fe-b98b-4e31-ac64-3b80da898040-kube-api-access-jm7k4\" (UniqueName: \"kubernetes.io/projected/dc24a7fe-b98b-4e31-ac64-3b80da898040-kube-api-access-jm7k4\") pod \"dc24a7fe-b98b-4e31-ac64-3b80da898040\" (UID: \"dc24a7fe-b98b-4e31-ac64-3b80da898040\") "
Oct 30 20:59:32.972955 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:32.971810    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc24a7fe-b98b-4e31-ac64-3b80da898040-plugins-config-volume" pod "dc24a7fe-b98b-4e31-ac64-3b80da898040" (UID: "dc24a7fe-b98b-4e31-ac64-3b80da898040"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:32.974237 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:32.974195    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc24a7fe-b98b-4e31-ac64-3b80da898040-kube-api-access-jm7k4" pod "dc24a7fe-b98b-4e31-ac64-3b80da898040" (UID: "dc24a7fe-b98b-4e31-ac64-3b80da898040"). InnerVolumeSpecName "kube-api-access-jm7k4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:33.072053 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:33.072011    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm7k4\" (UniqueName: \"kubernetes.io/projected/dc24a7fe-b98b-4e31-ac64-3b80da898040-kube-api-access-jm7k4\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:33.072053 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:33.072052    5071 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc24a7fe-b98b-4e31-ac64-3b80da898040-plugins-config-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:33.111765 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:33.111720    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:33.553481 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:33.553442    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:33.651421 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:33.651391    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-9e0fb1-7f7c8cbcc6-xzvm7" event={"ID":"d6cafaf5-279b-4ec5-b830-b27a7c00eb19","Type":"ContainerDied","Data":"92d75f355392807b9fcf28fd29ec5ee38d3707a5a8893db8dea824a8cd69bd2a"}
Oct 30 20:59:33.651421 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:33.651412    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-9e0fb1-7f7c8cbcc6-xzvm7"
Oct 30 20:59:33.651630 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:33.651462    5071 scope.go:122] "RemoveContainer" containerID="bdcd52dbb88e058054665f444178954745819b593e30fb45fcdc813685bd7346"
Oct 30 20:59:33.657407 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:33.657379    5071 generic.go:356] "Generic (PLEG): container finished" podID="dc24a7fe-b98b-4e31-ac64-3b80da898040" containerID="1b6642ae87b59e8a97266a4e379238b11b4563a49b0f5dde44127b399129a45b" exitCode=0
Oct 30 20:59:33.657547 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:33.657460    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-9e0fb1-78475c7fd7-gkjhq" event={"ID":"dc24a7fe-b98b-4e31-ac64-3b80da898040","Type":"ContainerDied","Data":"1b6642ae87b59e8a97266a4e379238b11b4563a49b0f5dde44127b399129a45b"}
Oct 30 20:59:33.657547 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:33.657501    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-9e0fb1-78475c7fd7-gkjhq" event={"ID":"dc24a7fe-b98b-4e31-ac64-3b80da898040","Type":"ContainerDied","Data":"0acc7392e26db3394c943628935670a514fd3933b5415b99411111bd1858c956"}
Oct 30 20:59:33.657671 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:33.657568    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-9e0fb1-78475c7fd7-gkjhq"
Oct 30 20:59:33.683473 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:33.682274    5071 scope.go:122] "RemoveContainer" containerID="1b6642ae87b59e8a97266a4e379238b11b4563a49b0f5dde44127b399129a45b"
Oct 30 20:59:33.713845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:33.712025    5071 scope.go:122] "RemoveContainer" containerID="1b6642ae87b59e8a97266a4e379238b11b4563a49b0f5dde44127b399129a45b"
Oct 30 20:59:33.751850 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:59:33.741389    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b6642ae87b59e8a97266a4e379238b11b4563a49b0f5dde44127b399129a45b\": container with ID starting with 1b6642ae87b59e8a97266a4e379238b11b4563a49b0f5dde44127b399129a45b not found: ID does not exist" containerID="1b6642ae87b59e8a97266a4e379238b11b4563a49b0f5dde44127b399129a45b"
Oct 30 20:59:33.751850 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:33.741434    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b6642ae87b59e8a97266a4e379238b11b4563a49b0f5dde44127b399129a45b"} err="failed to get container status \"1b6642ae87b59e8a97266a4e379238b11b4563a49b0f5dde44127b399129a45b\": rpc error: code = NotFound desc = could not find container \"1b6642ae87b59e8a97266a4e379238b11b4563a49b0f5dde44127b399129a45b\": container with ID starting with 1b6642ae87b59e8a97266a4e379238b11b4563a49b0f5dde44127b399129a45b not found: ID does not exist"
Oct 30 20:59:33.846813 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:33.846782    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-9e0fb1-7f7c8cbcc6-xzvm7"]
Oct 30 20:59:33.849470 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:33.849448    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-9e0fb1-7f7c8cbcc6-xzvm7"]
Oct 30 20:59:33.900993 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:33.900955    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:33.959113 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:33.959080    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-9e0fb1-78475c7fd7-gkjhq"]
Oct 30 20:59:33.967283 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:33.967253    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-9e0fb1-78475c7fd7-gkjhq"]
Oct 30 20:59:34.186449 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:34.186412    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:34.201652 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:34.201627    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6cafaf5-279b-4ec5-b830-b27a7c00eb19" path="/var/lib/kubelet/pods/d6cafaf5-279b-4ec5-b830-b27a7c00eb19/volumes"
Oct 30 20:59:34.202044 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:34.202030    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc24a7fe-b98b-4e31-ac64-3b80da898040" path="/var/lib/kubelet/pods/dc24a7fe-b98b-4e31-ac64-3b80da898040/volumes"
Oct 30 20:59:34.800462 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:34.800425    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:34.966331 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:34.966298    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:35.154282 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:35.154197    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:35.388188 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:35.388139    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:35.792384 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:35.792352    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:35.952048 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:35.952002    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:38.446932 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:38.446035    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:39.280625 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:39.280590    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:40.769571 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:40.769529    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:40.955119 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:40.955078    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:41.093767 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:41.093732    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"]
Oct 30 20:59:41.093991 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:41.093974    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d" podUID="5863cbee-17a0-4b2e-a6e5-c46737b72491" containerName="istio-proxy" containerID="cri-o://cdd1b09017184a821daf6d83c1295ebbe916122d53662895a7f6aee4c6f9878a" gracePeriod=30
Oct 30 20:59:41.125046 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:41.125019    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:41.476659 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:41.476569    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:42.028615 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:42.028581    5071 patch_prober.go:29] interesting pod/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:59:42.029038 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:42.028644    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d" podUID="5863cbee-17a0-4b2e-a6e5-c46737b72491" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:59:42.687082 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:42.687044    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x"]
Oct 30 20:59:42.687276 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:42.687251    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x" podUID="7312d39c-e785-458a-9ce8-53ee254ac19c" containerName="maas-api" containerID="cri-o://9bdf859ce16951794bacae438d4c895f0fa7ece2720ddcfe1bdf0ef75013ac76" gracePeriod=30
Oct 30 20:59:42.809027 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:42.808911    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:43.016169 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:43.016140    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x"
Oct 30 20:59:43.117445 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:43.117407    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7312d39c-e785-458a-9ce8-53ee254ac19c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7312d39c-e785-458a-9ce8-53ee254ac19c-openshift-service-ca\") pod \"7312d39c-e785-458a-9ce8-53ee254ac19c\" (UID: \"7312d39c-e785-458a-9ce8-53ee254ac19c\") "
Oct 30 20:59:43.118011 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:43.117491    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7312d39c-e785-458a-9ce8-53ee254ac19c-kube-api-access-nztn4\" (UniqueName: \"kubernetes.io/projected/7312d39c-e785-458a-9ce8-53ee254ac19c-kube-api-access-nztn4\") pod \"7312d39c-e785-458a-9ce8-53ee254ac19c\" (UID: \"7312d39c-e785-458a-9ce8-53ee254ac19c\") "
Oct 30 20:59:43.118011 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:43.117564    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7312d39c-e785-458a-9ce8-53ee254ac19c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7312d39c-e785-458a-9ce8-53ee254ac19c-maas-api-tls\") pod \"7312d39c-e785-458a-9ce8-53ee254ac19c\" (UID: \"7312d39c-e785-458a-9ce8-53ee254ac19c\") "
Oct 30 20:59:43.118011 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:43.117647    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7312d39c-e785-458a-9ce8-53ee254ac19c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7312d39c-e785-458a-9ce8-53ee254ac19c-metrics-certs\") pod \"7312d39c-e785-458a-9ce8-53ee254ac19c\" (UID: \"7312d39c-e785-458a-9ce8-53ee254ac19c\") "
Oct 30 20:59:43.118011 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:43.117862    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7312d39c-e785-458a-9ce8-53ee254ac19c-openshift-service-ca" pod "7312d39c-e785-458a-9ce8-53ee254ac19c" (UID: "7312d39c-e785-458a-9ce8-53ee254ac19c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:43.118499 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:43.118473    5071 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7312d39c-e785-458a-9ce8-53ee254ac19c-openshift-service-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:43.120317 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:43.120285    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7312d39c-e785-458a-9ce8-53ee254ac19c-metrics-certs" pod "7312d39c-e785-458a-9ce8-53ee254ac19c" (UID: "7312d39c-e785-458a-9ce8-53ee254ac19c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:59:43.120717 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:43.120690    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7312d39c-e785-458a-9ce8-53ee254ac19c-maas-api-tls" pod "7312d39c-e785-458a-9ce8-53ee254ac19c" (UID: "7312d39c-e785-458a-9ce8-53ee254ac19c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:59:43.120717 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:43.120695    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7312d39c-e785-458a-9ce8-53ee254ac19c-kube-api-access-nztn4" pod "7312d39c-e785-458a-9ce8-53ee254ac19c" (UID: "7312d39c-e785-458a-9ce8-53ee254ac19c"). InnerVolumeSpecName "kube-api-access-nztn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:43.220104 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:43.220069    5071 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7312d39c-e785-458a-9ce8-53ee254ac19c-metrics-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:43.220104 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:43.220099    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nztn4\" (UniqueName: \"kubernetes.io/projected/7312d39c-e785-458a-9ce8-53ee254ac19c-kube-api-access-nztn4\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:43.220104 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:43.220109    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7312d39c-e785-458a-9ce8-53ee254ac19c-maas-api-tls\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:43.728434 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:43.728394    5071 generic.go:356] "Generic (PLEG): container finished" podID="7312d39c-e785-458a-9ce8-53ee254ac19c" containerID="9bdf859ce16951794bacae438d4c895f0fa7ece2720ddcfe1bdf0ef75013ac76" exitCode=0
Oct 30 20:59:43.728652 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:43.728492    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x"
Oct 30 20:59:43.728652 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:43.728527    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x" event={"ID":"7312d39c-e785-458a-9ce8-53ee254ac19c","Type":"ContainerDied","Data":"9bdf859ce16951794bacae438d4c895f0fa7ece2720ddcfe1bdf0ef75013ac76"}
Oct 30 20:59:43.728652 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:43.728594    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x" event={"ID":"7312d39c-e785-458a-9ce8-53ee254ac19c","Type":"ContainerDied","Data":"3378d0224d2052fa64d37cc4aeb5bcaed851c972e288839d183d68609e476128"}
Oct 30 20:59:43.728652 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:43.728624    5071 scope.go:122] "RemoveContainer" containerID="9bdf859ce16951794bacae438d4c895f0fa7ece2720ddcfe1bdf0ef75013ac76"
Oct 30 20:59:43.740660 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:43.740638    5071 scope.go:122] "RemoveContainer" containerID="9bdf859ce16951794bacae438d4c895f0fa7ece2720ddcfe1bdf0ef75013ac76"
Oct 30 20:59:43.740957 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:59:43.740934    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bdf859ce16951794bacae438d4c895f0fa7ece2720ddcfe1bdf0ef75013ac76\": container with ID starting with 9bdf859ce16951794bacae438d4c895f0fa7ece2720ddcfe1bdf0ef75013ac76 not found: ID does not exist" containerID="9bdf859ce16951794bacae438d4c895f0fa7ece2720ddcfe1bdf0ef75013ac76"
Oct 30 20:59:43.741068 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:43.740961    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bdf859ce16951794bacae438d4c895f0fa7ece2720ddcfe1bdf0ef75013ac76"} err="failed to get container status \"9bdf859ce16951794bacae438d4c895f0fa7ece2720ddcfe1bdf0ef75013ac76\": rpc error: code = NotFound desc = could not find container \"9bdf859ce16951794bacae438d4c895f0fa7ece2720ddcfe1bdf0ef75013ac76\": container with ID starting with 9bdf859ce16951794bacae438d4c895f0fa7ece2720ddcfe1bdf0ef75013ac76 not found: ID does not exist"
Oct 30 20:59:43.894816 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:43.894780    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x"]
Oct 30 20:59:43.944751 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:43.944707    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-9e0fb1-bf7f6f746-fhr6x"]
Oct 30 20:59:44.201314 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:44.201279    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7312d39c-e785-458a-9ce8-53ee254ac19c" path="/var/lib/kubelet/pods/7312d39c-e785-458a-9ce8-53ee254ac19c/volumes"
Oct 30 20:59:44.368759 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:44.368729    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:44.586395 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:44.586361    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:44.742979 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:44.742946    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:45.303012 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:45.302972    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:46.369189 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.369121    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:59:46.455292 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.455264    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5863cbee-17a0-4b2e-a6e5-c46737b72491-kube-api-access-2zc2l\" (UniqueName: \"kubernetes.io/projected/5863cbee-17a0-4b2e-a6e5-c46737b72491-kube-api-access-2zc2l\") pod \"5863cbee-17a0-4b2e-a6e5-c46737b72491\" (UID: \"5863cbee-17a0-4b2e-a6e5-c46737b72491\") "
Oct 30 20:59:46.455471 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.455312    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5863cbee-17a0-4b2e-a6e5-c46737b72491-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5863cbee-17a0-4b2e-a6e5-c46737b72491-credential-socket\") pod \"5863cbee-17a0-4b2e-a6e5-c46737b72491\" (UID: \"5863cbee-17a0-4b2e-a6e5-c46737b72491\") "
Oct 30 20:59:46.455471 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.455442    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5863cbee-17a0-4b2e-a6e5-c46737b72491-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5863cbee-17a0-4b2e-a6e5-c46737b72491-istiod-ca-cert\") pod \"5863cbee-17a0-4b2e-a6e5-c46737b72491\" (UID: \"5863cbee-17a0-4b2e-a6e5-c46737b72491\") "
Oct 30 20:59:46.455604 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.455546    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5863cbee-17a0-4b2e-a6e5-c46737b72491-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5863cbee-17a0-4b2e-a6e5-c46737b72491-istio-data\") pod \"5863cbee-17a0-4b2e-a6e5-c46737b72491\" (UID: \"5863cbee-17a0-4b2e-a6e5-c46737b72491\") "
Oct 30 20:59:46.455604 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.455590    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5863cbee-17a0-4b2e-a6e5-c46737b72491-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5863cbee-17a0-4b2e-a6e5-c46737b72491-istio-envoy\") pod \"5863cbee-17a0-4b2e-a6e5-c46737b72491\" (UID: \"5863cbee-17a0-4b2e-a6e5-c46737b72491\") "
Oct 30 20:59:46.455744 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.455655    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5863cbee-17a0-4b2e-a6e5-c46737b72491-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5863cbee-17a0-4b2e-a6e5-c46737b72491-workload-socket\") pod \"5863cbee-17a0-4b2e-a6e5-c46737b72491\" (UID: \"5863cbee-17a0-4b2e-a6e5-c46737b72491\") "
Oct 30 20:59:46.455744 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.455636    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5863cbee-17a0-4b2e-a6e5-c46737b72491-credential-socket" pod "5863cbee-17a0-4b2e-a6e5-c46737b72491" (UID: "5863cbee-17a0-4b2e-a6e5-c46737b72491"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:59:46.455744 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.455688    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5863cbee-17a0-4b2e-a6e5-c46737b72491-istio-token\" (UniqueName: \"kubernetes.io/projected/5863cbee-17a0-4b2e-a6e5-c46737b72491-istio-token\") pod \"5863cbee-17a0-4b2e-a6e5-c46737b72491\" (UID: \"5863cbee-17a0-4b2e-a6e5-c46737b72491\") "
Oct 30 20:59:46.455943 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.455790    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5863cbee-17a0-4b2e-a6e5-c46737b72491-istiod-ca-cert" pod "5863cbee-17a0-4b2e-a6e5-c46737b72491" (UID: "5863cbee-17a0-4b2e-a6e5-c46737b72491"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:46.455943 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.455795    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5863cbee-17a0-4b2e-a6e5-c46737b72491-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5863cbee-17a0-4b2e-a6e5-c46737b72491-istio-podinfo\") pod \"5863cbee-17a0-4b2e-a6e5-c46737b72491\" (UID: \"5863cbee-17a0-4b2e-a6e5-c46737b72491\") "
Oct 30 20:59:46.455943 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.455894    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5863cbee-17a0-4b2e-a6e5-c46737b72491-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5863cbee-17a0-4b2e-a6e5-c46737b72491-workload-certs\") pod \"5863cbee-17a0-4b2e-a6e5-c46737b72491\" (UID: \"5863cbee-17a0-4b2e-a6e5-c46737b72491\") "
Oct 30 20:59:46.456716 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.456022    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5863cbee-17a0-4b2e-a6e5-c46737b72491-istio-data" pod "5863cbee-17a0-4b2e-a6e5-c46737b72491" (UID: "5863cbee-17a0-4b2e-a6e5-c46737b72491"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:59:46.456716 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.456232    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5863cbee-17a0-4b2e-a6e5-c46737b72491-workload-socket" pod "5863cbee-17a0-4b2e-a6e5-c46737b72491" (UID: "5863cbee-17a0-4b2e-a6e5-c46737b72491"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:59:46.456716 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.456407    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5863cbee-17a0-4b2e-a6e5-c46737b72491-workload-certs" pod "5863cbee-17a0-4b2e-a6e5-c46737b72491" (UID: "5863cbee-17a0-4b2e-a6e5-c46737b72491"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:59:46.456716 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.456683    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5863cbee-17a0-4b2e-a6e5-c46737b72491-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:46.456716 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.456708    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5863cbee-17a0-4b2e-a6e5-c46737b72491-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:46.457115 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.456722    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5863cbee-17a0-4b2e-a6e5-c46737b72491-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:46.457115 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.456738    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5863cbee-17a0-4b2e-a6e5-c46737b72491-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:46.457115 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.456752    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5863cbee-17a0-4b2e-a6e5-c46737b72491-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:46.458171 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.458149    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5863cbee-17a0-4b2e-a6e5-c46737b72491-istio-envoy" pod "5863cbee-17a0-4b2e-a6e5-c46737b72491" (UID: "5863cbee-17a0-4b2e-a6e5-c46737b72491"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:59:46.458276 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.458197    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5863cbee-17a0-4b2e-a6e5-c46737b72491-istio-podinfo" pod "5863cbee-17a0-4b2e-a6e5-c46737b72491" (UID: "5863cbee-17a0-4b2e-a6e5-c46737b72491"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:59:46.458432 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.458418    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5863cbee-17a0-4b2e-a6e5-c46737b72491-kube-api-access-2zc2l" pod "5863cbee-17a0-4b2e-a6e5-c46737b72491" (UID: "5863cbee-17a0-4b2e-a6e5-c46737b72491"). InnerVolumeSpecName "kube-api-access-2zc2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:46.458628 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.458609    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5863cbee-17a0-4b2e-a6e5-c46737b72491-istio-token" pod "5863cbee-17a0-4b2e-a6e5-c46737b72491" (UID: "5863cbee-17a0-4b2e-a6e5-c46737b72491"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:46.558379 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.558340    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5863cbee-17a0-4b2e-a6e5-c46737b72491-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:46.558379 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.558368    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5863cbee-17a0-4b2e-a6e5-c46737b72491-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:46.558379 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.558379    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5863cbee-17a0-4b2e-a6e5-c46737b72491-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:46.558695 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.558391    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zc2l\" (UniqueName: \"kubernetes.io/projected/5863cbee-17a0-4b2e-a6e5-c46737b72491-kube-api-access-2zc2l\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:46.727647 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:59:46.727556    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 20:59:46.753937 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.753900    5071 generic.go:356] "Generic (PLEG): container finished" podID="5863cbee-17a0-4b2e-a6e5-c46737b72491" containerID="cdd1b09017184a821daf6d83c1295ebbe916122d53662895a7f6aee4c6f9878a" exitCode=0
Oct 30 20:59:46.754104 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.753966    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"
Oct 30 20:59:46.754104 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.753980    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d" event={"ID":"5863cbee-17a0-4b2e-a6e5-c46737b72491","Type":"ContainerDied","Data":"cdd1b09017184a821daf6d83c1295ebbe916122d53662895a7f6aee4c6f9878a"}
Oct 30 20:59:46.754104 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.754036    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d" event={"ID":"5863cbee-17a0-4b2e-a6e5-c46737b72491","Type":"ContainerDied","Data":"af00e45e344752dbd2b3a62822999c705645ccfa620a539342d9acdceb744cb2"}
Oct 30 20:59:46.754104 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.754060    5071 scope.go:122] "RemoveContainer" containerID="cdd1b09017184a821daf6d83c1295ebbe916122d53662895a7f6aee4c6f9878a"
Oct 30 20:59:46.765089 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.765069    5071 scope.go:122] "RemoveContainer" containerID="fab3657d130ceced6ecc5f39b5a6707979abc9d0738657294ab14ba60df5fe37"
Oct 30 20:59:46.774504 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.774482    5071 scope.go:122] "RemoveContainer" containerID="cdd1b09017184a821daf6d83c1295ebbe916122d53662895a7f6aee4c6f9878a"
Oct 30 20:59:46.774780 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:59:46.774753    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdd1b09017184a821daf6d83c1295ebbe916122d53662895a7f6aee4c6f9878a\": container with ID starting with cdd1b09017184a821daf6d83c1295ebbe916122d53662895a7f6aee4c6f9878a not found: ID does not exist" containerID="cdd1b09017184a821daf6d83c1295ebbe916122d53662895a7f6aee4c6f9878a"
Oct 30 20:59:46.774866 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.774791    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd1b09017184a821daf6d83c1295ebbe916122d53662895a7f6aee4c6f9878a"} err="failed to get container status \"cdd1b09017184a821daf6d83c1295ebbe916122d53662895a7f6aee4c6f9878a\": rpc error: code = NotFound desc = could not find container \"cdd1b09017184a821daf6d83c1295ebbe916122d53662895a7f6aee4c6f9878a\": container with ID starting with cdd1b09017184a821daf6d83c1295ebbe916122d53662895a7f6aee4c6f9878a not found: ID does not exist"
Oct 30 20:59:46.774866 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.774847    5071 scope.go:122] "RemoveContainer" containerID="fab3657d130ceced6ecc5f39b5a6707979abc9d0738657294ab14ba60df5fe37"
Oct 30 20:59:46.775119 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:59:46.775099    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fab3657d130ceced6ecc5f39b5a6707979abc9d0738657294ab14ba60df5fe37\": container with ID starting with fab3657d130ceced6ecc5f39b5a6707979abc9d0738657294ab14ba60df5fe37 not found: ID does not exist" containerID="fab3657d130ceced6ecc5f39b5a6707979abc9d0738657294ab14ba60df5fe37"
Oct 30 20:59:46.775167 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.775127    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fab3657d130ceced6ecc5f39b5a6707979abc9d0738657294ab14ba60df5fe37"} err="failed to get container status \"fab3657d130ceced6ecc5f39b5a6707979abc9d0738657294ab14ba60df5fe37\": rpc error: code = NotFound desc = could not find container \"fab3657d130ceced6ecc5f39b5a6707979abc9d0738657294ab14ba60df5fe37\": container with ID starting with fab3657d130ceced6ecc5f39b5a6707979abc9d0738657294ab14ba60df5fe37 not found: ID does not exist"
Oct 30 20:59:46.920771 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.920737    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"]
Oct 30 20:59:46.970482 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:46.970404    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b43c84-openshift-default-7b7d7b556f-jhx5d"]
Oct 30 20:59:48.202289 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:48.202246    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5863cbee-17a0-4b2e-a6e5-c46737b72491" path="/var/lib/kubelet/pods/5863cbee-17a0-4b2e-a6e5-c46737b72491/volumes"
Oct 30 20:59:49.021516 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:49.021481    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w"]
Oct 30 20:59:49.021755 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:49.021720    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w" podUID="e830ac30-2863-47d3-a501-5d0622dccb81" containerName="maas-api" containerID="cri-o://e94670734adcd150dffab1ec688fdc4e7e1b4cd1b1a46ffc37aaab998c7530a0" gracePeriod=30
Oct 30 20:59:49.141698 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:49.141648    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:49.330340 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:49.330312    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w"
Oct 30 20:59:49.420475 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:49.418490    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e830ac30-2863-47d3-a501-5d0622dccb81-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e830ac30-2863-47d3-a501-5d0622dccb81-metrics-certs\") pod \"e830ac30-2863-47d3-a501-5d0622dccb81\" (UID: \"e830ac30-2863-47d3-a501-5d0622dccb81\") "
Oct 30 20:59:49.420475 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:49.418621    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e830ac30-2863-47d3-a501-5d0622dccb81-kube-api-access-c628k\" (UniqueName: \"kubernetes.io/projected/e830ac30-2863-47d3-a501-5d0622dccb81-kube-api-access-c628k\") pod \"e830ac30-2863-47d3-a501-5d0622dccb81\" (UID: \"e830ac30-2863-47d3-a501-5d0622dccb81\") "
Oct 30 20:59:49.420475 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:49.418662    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e830ac30-2863-47d3-a501-5d0622dccb81-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e830ac30-2863-47d3-a501-5d0622dccb81-maas-api-tls\") pod \"e830ac30-2863-47d3-a501-5d0622dccb81\" (UID: \"e830ac30-2863-47d3-a501-5d0622dccb81\") "
Oct 30 20:59:49.420475 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:49.418712    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e830ac30-2863-47d3-a501-5d0622dccb81-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e830ac30-2863-47d3-a501-5d0622dccb81-openshift-service-ca\") pod \"e830ac30-2863-47d3-a501-5d0622dccb81\" (UID: \"e830ac30-2863-47d3-a501-5d0622dccb81\") "
Oct 30 20:59:49.426433 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:49.426370    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e830ac30-2863-47d3-a501-5d0622dccb81-maas-api-tls" pod "e830ac30-2863-47d3-a501-5d0622dccb81" (UID: "e830ac30-2863-47d3-a501-5d0622dccb81"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:59:49.426622 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:49.426522    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e830ac30-2863-47d3-a501-5d0622dccb81-kube-api-access-c628k" pod "e830ac30-2863-47d3-a501-5d0622dccb81" (UID: "e830ac30-2863-47d3-a501-5d0622dccb81"). InnerVolumeSpecName "kube-api-access-c628k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:49.427132 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:49.427001    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e830ac30-2863-47d3-a501-5d0622dccb81-openshift-service-ca" pod "e830ac30-2863-47d3-a501-5d0622dccb81" (UID: "e830ac30-2863-47d3-a501-5d0622dccb81"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:49.438012 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:49.437953    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e830ac30-2863-47d3-a501-5d0622dccb81-metrics-certs" pod "e830ac30-2863-47d3-a501-5d0622dccb81" (UID: "e830ac30-2863-47d3-a501-5d0622dccb81"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:59:49.520016 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:49.519983    5071 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e830ac30-2863-47d3-a501-5d0622dccb81-metrics-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:49.520016 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:49.520012    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c628k\" (UniqueName: \"kubernetes.io/projected/e830ac30-2863-47d3-a501-5d0622dccb81-kube-api-access-c628k\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:49.520016 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:49.520023    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e830ac30-2863-47d3-a501-5d0622dccb81-maas-api-tls\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:49.520339 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:49.520033    5071 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e830ac30-2863-47d3-a501-5d0622dccb81-openshift-service-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:49.776794 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:49.776690    5071 generic.go:356] "Generic (PLEG): container finished" podID="e830ac30-2863-47d3-a501-5d0622dccb81" containerID="e94670734adcd150dffab1ec688fdc4e7e1b4cd1b1a46ffc37aaab998c7530a0" exitCode=0
Oct 30 20:59:49.776794 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:49.776751    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w"
Oct 30 20:59:49.776794 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:49.776772    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w" event={"ID":"e830ac30-2863-47d3-a501-5d0622dccb81","Type":"ContainerDied","Data":"e94670734adcd150dffab1ec688fdc4e7e1b4cd1b1a46ffc37aaab998c7530a0"}
Oct 30 20:59:49.777121 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:49.776878    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w" event={"ID":"e830ac30-2863-47d3-a501-5d0622dccb81","Type":"ContainerDied","Data":"c56249508f3922492ccae759f5229982a635748b482cae119588c92e7661441c"}
Oct 30 20:59:49.777121 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:49.776908    5071 scope.go:122] "RemoveContainer" containerID="e94670734adcd150dffab1ec688fdc4e7e1b4cd1b1a46ffc37aaab998c7530a0"
Oct 30 20:59:49.788014 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:49.787989    5071 scope.go:122] "RemoveContainer" containerID="e94670734adcd150dffab1ec688fdc4e7e1b4cd1b1a46ffc37aaab998c7530a0"
Oct 30 20:59:49.788292 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:59:49.788275    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e94670734adcd150dffab1ec688fdc4e7e1b4cd1b1a46ffc37aaab998c7530a0\": container with ID starting with e94670734adcd150dffab1ec688fdc4e7e1b4cd1b1a46ffc37aaab998c7530a0 not found: ID does not exist" containerID="e94670734adcd150dffab1ec688fdc4e7e1b4cd1b1a46ffc37aaab998c7530a0"
Oct 30 20:59:49.788351 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:49.788300    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e94670734adcd150dffab1ec688fdc4e7e1b4cd1b1a46ffc37aaab998c7530a0"} err="failed to get container status \"e94670734adcd150dffab1ec688fdc4e7e1b4cd1b1a46ffc37aaab998c7530a0\": rpc error: code = NotFound desc = could not find container \"e94670734adcd150dffab1ec688fdc4e7e1b4cd1b1a46ffc37aaab998c7530a0\": container with ID starting with e94670734adcd150dffab1ec688fdc4e7e1b4cd1b1a46ffc37aaab998c7530a0 not found: ID does not exist"
Oct 30 20:59:49.949845 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:49.948197    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w"]
Oct 30 20:59:49.962147 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:49.952338    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d9b3f6-7cf865b976-h952w"]
Oct 30 20:59:50.201075 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:50.201041    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e830ac30-2863-47d3-a501-5d0622dccb81" path="/var/lib/kubelet/pods/e830ac30-2863-47d3-a501-5d0622dccb81/volumes"
Oct 30 20:59:50.698111 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:50.698076    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:50.998550 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:50.998462    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:51.156551 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:51.156516    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:51.631230 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:51.631191    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:52.121213 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:52.121176    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 20:59:53.700399 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:53.700358    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:53.879282 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:53.879249    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:54.050094 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:54.050062    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"]
Oct 30 20:59:54.050286 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:54.050265    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz" podUID="7e3d8feb-58a2-40e0-b5b1-94acb06fe606" containerName="istio-proxy" containerID="cri-o://b83be4defe683183511b6df6616f51c17dfcffb8887f8aa3ca17748cb77a761b" gracePeriod=30
Oct 30 20:59:54.188796 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:54.188760    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:54.581316 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:54.581274    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:55.050668 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.050626    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee3fad-pjr8f"]
Oct 30 20:59:55.051183 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.051169    5071 cpu_manager.go:414] "containerMap: removing container" podUID="5863cbee-17a0-4b2e-a6e5-c46737b72491" containerName="istio-proxy"
Oct 30 20:59:55.051224 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.051185    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5863cbee-17a0-4b2e-a6e5-c46737b72491" containerName="istio-proxy"
Oct 30 20:59:55.051224 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.051199    5071 cpu_manager.go:414] "containerMap: removing container" podUID="b31f6255-ed4b-45af-9a92-e2de6f5ee723" containerName="maas-api"
Oct 30 20:59:55.051224 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.051206    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b31f6255-ed4b-45af-9a92-e2de6f5ee723" containerName="maas-api"
Oct 30 20:59:55.051337 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.051237    5071 cpu_manager.go:414] "containerMap: removing container" podUID="e830ac30-2863-47d3-a501-5d0622dccb81" containerName="maas-api"
Oct 30 20:59:55.051337 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.051247    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e830ac30-2863-47d3-a501-5d0622dccb81" containerName="maas-api"
Oct 30 20:59:55.051337 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.051261    5071 cpu_manager.go:414] "containerMap: removing container" podUID="7312d39c-e785-458a-9ce8-53ee254ac19c" containerName="maas-api"
Oct 30 20:59:55.051337 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.051268    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7312d39c-e785-458a-9ce8-53ee254ac19c" containerName="maas-api"
Oct 30 20:59:55.051337 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.051288    5071 cpu_manager.go:414] "containerMap: removing container" podUID="715ab59c-989b-4cfb-bfa7-77ca2c041e9d" containerName="test-curl-7209-4450a6"
Oct 30 20:59:55.051337 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.051293    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="715ab59c-989b-4cfb-bfa7-77ca2c041e9d" containerName="test-curl-7209-4450a6"
Oct 30 20:59:55.051337 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.051303    5071 cpu_manager.go:414] "containerMap: removing container" podUID="1beb9127-eb1c-41cf-8217-52ec9d5d258c" containerName="payload-pre-processing"
Oct 30 20:59:55.051337 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.051308    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1beb9127-eb1c-41cf-8217-52ec9d5d258c" containerName="payload-pre-processing"
Oct 30 20:59:55.051337 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.051317    5071 cpu_manager.go:414] "containerMap: removing container" podUID="5863cbee-17a0-4b2e-a6e5-c46737b72491" containerName="istio-proxy"
Oct 30 20:59:55.051337 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.051322    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5863cbee-17a0-4b2e-a6e5-c46737b72491" containerName="istio-proxy"
Oct 30 20:59:55.051337 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.051333    5071 cpu_manager.go:414] "containerMap: removing container" podUID="49cfe5da-09b4-4268-8938-4d6f10827d35" containerName="payload-processing"
Oct 30 20:59:55.051337 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.051338    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49cfe5da-09b4-4268-8938-4d6f10827d35" containerName="payload-processing"
Oct 30 20:59:55.051685 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.051346    5071 cpu_manager.go:414] "containerMap: removing container" podUID="dc24a7fe-b98b-4e31-ac64-3b80da898040" containerName="payload-pre-processing"
Oct 30 20:59:55.051685 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.051350    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc24a7fe-b98b-4e31-ac64-3b80da898040" containerName="payload-pre-processing"
Oct 30 20:59:55.051685 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.051360    5071 cpu_manager.go:414] "containerMap: removing container" podUID="d6cafaf5-279b-4ec5-b830-b27a7c00eb19" containerName="payload-processing"
Oct 30 20:59:55.051685 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.051366    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6cafaf5-279b-4ec5-b830-b27a7c00eb19" containerName="payload-processing"
Oct 30 20:59:55.051685 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.051374    5071 cpu_manager.go:414] "containerMap: removing container" podUID="940a58d0-5deb-4bc9-8832-5a17fa300539" containerName="payload-pre-processing"
Oct 30 20:59:55.051685 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.051379    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="940a58d0-5deb-4bc9-8832-5a17fa300539" containerName="payload-pre-processing"
Oct 30 20:59:55.051685 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.051385    5071 cpu_manager.go:414] "containerMap: removing container" podUID="902523f4-4967-4ef2-97be-632d3a0c6383" containerName="payload-processing"
Oct 30 20:59:55.051685 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.051390    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="902523f4-4967-4ef2-97be-632d3a0c6383" containerName="payload-processing"
Oct 30 20:59:55.051685 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.051471    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="49cfe5da-09b4-4268-8938-4d6f10827d35" containerName="payload-processing"
Oct 30 20:59:55.051685 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.051482    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc24a7fe-b98b-4e31-ac64-3b80da898040" containerName="payload-pre-processing"
Oct 30 20:59:55.051685 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.051489    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6cafaf5-279b-4ec5-b830-b27a7c00eb19" containerName="payload-processing"
Oct 30 20:59:55.051685 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.051495    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="940a58d0-5deb-4bc9-8832-5a17fa300539" containerName="payload-pre-processing"
Oct 30 20:59:55.051685 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.051501    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="902523f4-4967-4ef2-97be-632d3a0c6383" containerName="payload-processing"
Oct 30 20:59:55.051685 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.051506    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="5863cbee-17a0-4b2e-a6e5-c46737b72491" containerName="istio-proxy"
Oct 30 20:59:55.051685 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.051511    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="b31f6255-ed4b-45af-9a92-e2de6f5ee723" containerName="maas-api"
Oct 30 20:59:55.051685 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.051518    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="e830ac30-2863-47d3-a501-5d0622dccb81" containerName="maas-api"
Oct 30 20:59:55.051685 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.051523    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="7312d39c-e785-458a-9ce8-53ee254ac19c" containerName="maas-api"
Oct 30 20:59:55.051685 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.051530    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="715ab59c-989b-4cfb-bfa7-77ca2c041e9d" containerName="test-curl-7209-4450a6"
Oct 30 20:59:55.051685 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.051536    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="1beb9127-eb1c-41cf-8217-52ec9d5d258c" containerName="payload-pre-processing"
Oct 30 20:59:55.056362 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.056342    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee3fad-pjr8f"
Oct 30 20:59:55.101686 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.101659    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57013dd4-1ed6-4ebf-a118-c5ea00b8c087-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-ee3fad-pjr8f\" (UID: \"57013dd4-1ed6-4ebf-a118-c5ea00b8c087\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee3fad-pjr8f"
Oct 30 20:59:55.101789 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.101702    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ztlb\" (UniqueName: \"kubernetes.io/projected/57013dd4-1ed6-4ebf-a118-c5ea00b8c087-kube-api-access-7ztlb\") pod \"maas-api-revoke-keys-e2e-worker-w6-ee3fad-pjr8f\" (UID: \"57013dd4-1ed6-4ebf-a118-c5ea00b8c087\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee3fad-pjr8f"
Oct 30 20:59:55.101890 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.101861    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57013dd4-1ed6-4ebf-a118-c5ea00b8c087-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-ee3fad-pjr8f\" (UID: \"57013dd4-1ed6-4ebf-a118-c5ea00b8c087\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee3fad-pjr8f"
Oct 30 20:59:55.113278 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.113254    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee3fad-pjr8f"]
Oct 30 20:59:55.159394 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.159363    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:59:55.159546 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.159444    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 20:59:55.202575 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.202538    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57013dd4-1ed6-4ebf-a118-c5ea00b8c087-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-ee3fad-pjr8f\" (UID: \"57013dd4-1ed6-4ebf-a118-c5ea00b8c087\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee3fad-pjr8f"
Oct 30 20:59:55.202772 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.202593    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ztlb\" (UniqueName: \"kubernetes.io/projected/57013dd4-1ed6-4ebf-a118-c5ea00b8c087-kube-api-access-7ztlb\") pod \"maas-api-revoke-keys-e2e-worker-w6-ee3fad-pjr8f\" (UID: \"57013dd4-1ed6-4ebf-a118-c5ea00b8c087\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee3fad-pjr8f"
Oct 30 20:59:55.202772 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.202756    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57013dd4-1ed6-4ebf-a118-c5ea00b8c087-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-ee3fad-pjr8f\" (UID: \"57013dd4-1ed6-4ebf-a118-c5ea00b8c087\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee3fad-pjr8f"
Oct 30 20:59:55.203425 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.203399    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57013dd4-1ed6-4ebf-a118-c5ea00b8c087-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-ee3fad-pjr8f\" (UID: \"57013dd4-1ed6-4ebf-a118-c5ea00b8c087\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee3fad-pjr8f"
Oct 30 20:59:55.205252 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.205233    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57013dd4-1ed6-4ebf-a118-c5ea00b8c087-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-ee3fad-pjr8f\" (UID: \"57013dd4-1ed6-4ebf-a118-c5ea00b8c087\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee3fad-pjr8f"
Oct 30 20:59:55.260737 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.260706    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ztlb\" (UniqueName: \"kubernetes.io/projected/57013dd4-1ed6-4ebf-a118-c5ea00b8c087-kube-api-access-7ztlb\") pod \"maas-api-revoke-keys-e2e-worker-w6-ee3fad-pjr8f\" (UID: \"57013dd4-1ed6-4ebf-a118-c5ea00b8c087\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee3fad-pjr8f"
Oct 30 20:59:55.369706 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.369627    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee3fad-pjr8f"
Oct 30 20:59:55.756695 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.756661    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee3fad-pjr8f"]
Oct 30 20:59:55.757966 ip-10-0-139-149 kubenswrapper[5071]: W1030 20:59:55.757941    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57013dd4_1ed6_4ebf_a118_c5ea00b8c087.slice/crio-e1a04bb013a27bcacf8b42d218a1bad6008e526bf591115152e16c8f615311ba WatchSource:0}: Error finding container e1a04bb013a27bcacf8b42d218a1bad6008e526bf591115152e16c8f615311ba: Status 404 returned error can't find the container with id e1a04bb013a27bcacf8b42d218a1bad6008e526bf591115152e16c8f615311ba
Oct 30 20:59:55.816673 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:55.816642    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee3fad-pjr8f" event={"ID":"57013dd4-1ed6-4ebf-a118-c5ea00b8c087","Type":"ContainerStarted","Data":"e1a04bb013a27bcacf8b42d218a1bad6008e526bf591115152e16c8f615311ba"}
Oct 30 20:59:56.281751 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:56.281711    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:56.825085 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:56.825049    5071 generic.go:356] "Generic (PLEG): container finished" podID="57013dd4-1ed6-4ebf-a118-c5ea00b8c087" containerID="f9c32a58e76435c65e228f3e14d6e9c3f2fc233ef4bcc88168054f00c5a89280" exitCode=0
Oct 30 20:59:56.825266 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:56.825133    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee3fad-pjr8f" event={"ID":"57013dd4-1ed6-4ebf-a118-c5ea00b8c087","Type":"ContainerDied","Data":"f9c32a58e76435c65e228f3e14d6e9c3f2fc233ef4bcc88168054f00c5a89280"}
Oct 30 20:59:57.629217 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:57.629181    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:57.850915 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:57.850859    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:57.996832 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:57.996799    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee3fad-pjr8f"
Oct 30 20:59:58.031499 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:58.031475    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57013dd4-1ed6-4ebf-a118-c5ea00b8c087-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57013dd4-1ed6-4ebf-a118-c5ea00b8c087-maas-api-openshift-ca\") pod \"57013dd4-1ed6-4ebf-a118-c5ea00b8c087\" (UID: \"57013dd4-1ed6-4ebf-a118-c5ea00b8c087\") "
Oct 30 20:59:58.031639 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:58.031545    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57013dd4-1ed6-4ebf-a118-c5ea00b8c087-kube-api-access-7ztlb\" (UniqueName: \"kubernetes.io/projected/57013dd4-1ed6-4ebf-a118-c5ea00b8c087-kube-api-access-7ztlb\") pod \"57013dd4-1ed6-4ebf-a118-c5ea00b8c087\" (UID: \"57013dd4-1ed6-4ebf-a118-c5ea00b8c087\") "
Oct 30 20:59:58.031639 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:58.031580    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57013dd4-1ed6-4ebf-a118-c5ea00b8c087-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57013dd4-1ed6-4ebf-a118-c5ea00b8c087-maas-api-xks-ca\") pod \"57013dd4-1ed6-4ebf-a118-c5ea00b8c087\" (UID: \"57013dd4-1ed6-4ebf-a118-c5ea00b8c087\") "
Oct 30 20:59:58.032148 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:58.031887    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57013dd4-1ed6-4ebf-a118-c5ea00b8c087-maas-api-openshift-ca" pod "57013dd4-1ed6-4ebf-a118-c5ea00b8c087" (UID: "57013dd4-1ed6-4ebf-a118-c5ea00b8c087"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:58.032148 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:58.032110    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57013dd4-1ed6-4ebf-a118-c5ea00b8c087-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:58.036860 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:58.034391    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57013dd4-1ed6-4ebf-a118-c5ea00b8c087-maas-api-xks-ca" pod "57013dd4-1ed6-4ebf-a118-c5ea00b8c087" (UID: "57013dd4-1ed6-4ebf-a118-c5ea00b8c087"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:59:58.036860 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:58.034417    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57013dd4-1ed6-4ebf-a118-c5ea00b8c087-kube-api-access-7ztlb" pod "57013dd4-1ed6-4ebf-a118-c5ea00b8c087" (UID: "57013dd4-1ed6-4ebf-a118-c5ea00b8c087"). InnerVolumeSpecName "kube-api-access-7ztlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:58.133140 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:58.133101    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ztlb\" (UniqueName: \"kubernetes.io/projected/57013dd4-1ed6-4ebf-a118-c5ea00b8c087-kube-api-access-7ztlb\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:58.133140 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:58.133129    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57013dd4-1ed6-4ebf-a118-c5ea00b8c087-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:58.567540 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:58.567491    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:58.775011 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:58.774974    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:58.844677 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:58.844590    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee3fad-pjr8f"
Oct 30 20:59:58.844892 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:58.844584    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee3fad-pjr8f" event={"ID":"57013dd4-1ed6-4ebf-a118-c5ea00b8c087","Type":"ContainerDied","Data":"e1a04bb013a27bcacf8b42d218a1bad6008e526bf591115152e16c8f615311ba"}
Oct 30 20:59:58.844892 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:58.844760    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1a04bb013a27bcacf8b42d218a1bad6008e526bf591115152e16c8f615311ba"
Oct 30 20:59:58.910701 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:58.910665    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"]
Oct 30 20:59:58.910954 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:58.910931    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw" podUID="2c36da32-7ded-4d53-8579-f62445fc61a5" containerName="istio-proxy" containerID="cri-o://15cdf37757674bf07863987b38358fb28ad8564879f10999a9c3ca279dcdf069" gracePeriod=30
Oct 30 20:59:58.936676 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:58.936640    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:59.335014 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.334987    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:59:59.449435 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.449350    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-kube-api-access-kq97w\" (UniqueName: \"kubernetes.io/projected/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-kube-api-access-kq97w\") pod \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\" (UID: \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\") "
Oct 30 20:59:59.449613 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.449433    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-istio-envoy\") pod \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\" (UID: \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\") "
Oct 30 20:59:59.449613 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.449474    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-istio-data\") pod \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\" (UID: \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\") "
Oct 30 20:59:59.449613 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.449503    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-istio-token\" (UniqueName: \"kubernetes.io/projected/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-istio-token\") pod \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\" (UID: \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\") "
Oct 30 20:59:59.449613 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.449582    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-istio-podinfo\") pod \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\" (UID: \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\") "
Oct 30 20:59:59.449873 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.449652    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-workload-certs\") pod \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\" (UID: \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\") "
Oct 30 20:59:59.449873 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.449728    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-workload-socket\") pod \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\" (UID: \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\") "
Oct 30 20:59:59.449873 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.449775    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-istio-data" pod "7e3d8feb-58a2-40e0-b5b1-94acb06fe606" (UID: "7e3d8feb-58a2-40e0-b5b1-94acb06fe606"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:59:59.449873 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.449804    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-credential-socket\") pod \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\" (UID: \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\") "
Oct 30 20:59:59.450081 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.449887    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-istiod-ca-cert\") pod \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\" (UID: \"7e3d8feb-58a2-40e0-b5b1-94acb06fe606\") "
Oct 30 20:59:59.450153 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.450126    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-workload-socket" pod "7e3d8feb-58a2-40e0-b5b1-94acb06fe606" (UID: "7e3d8feb-58a2-40e0-b5b1-94acb06fe606"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:59:59.450440 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.450303    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-workload-certs" pod "7e3d8feb-58a2-40e0-b5b1-94acb06fe606" (UID: "7e3d8feb-58a2-40e0-b5b1-94acb06fe606"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:59:59.450440 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.450353    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:59.450440 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.450374    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:59.450704 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.450515    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-credential-socket" pod "7e3d8feb-58a2-40e0-b5b1-94acb06fe606" (UID: "7e3d8feb-58a2-40e0-b5b1-94acb06fe606"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:59:59.450704 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.450579    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-istiod-ca-cert" pod "7e3d8feb-58a2-40e0-b5b1-94acb06fe606" (UID: "7e3d8feb-58a2-40e0-b5b1-94acb06fe606"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:59.452524 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.452454    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-istio-envoy" pod "7e3d8feb-58a2-40e0-b5b1-94acb06fe606" (UID: "7e3d8feb-58a2-40e0-b5b1-94acb06fe606"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:59:59.453614 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.453457    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-istio-podinfo" pod "7e3d8feb-58a2-40e0-b5b1-94acb06fe606" (UID: "7e3d8feb-58a2-40e0-b5b1-94acb06fe606"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:59:59.453614 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.453559    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-kube-api-access-kq97w" pod "7e3d8feb-58a2-40e0-b5b1-94acb06fe606" (UID: "7e3d8feb-58a2-40e0-b5b1-94acb06fe606"). InnerVolumeSpecName "kube-api-access-kq97w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:59.453614 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.453591    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-istio-token" pod "7e3d8feb-58a2-40e0-b5b1-94acb06fe606" (UID: "7e3d8feb-58a2-40e0-b5b1-94acb06fe606"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:59.551754 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.551716    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:59.551754 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.551758    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:59.551983 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.551775    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:59.551983 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.551789    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:59.551983 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.551802    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:59.551983 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.551815    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq97w\" (UniqueName: \"kubernetes.io/projected/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-kube-api-access-kq97w\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:59.551983 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.551867    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e3d8feb-58a2-40e0-b5b1-94acb06fe606-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:59.649262 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.649226    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 20:59:59.788541 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.788508    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-19176d-fwph5"]
Oct 30 20:59:59.789300 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.789277    5071 cpu_manager.go:414] "containerMap: removing container" podUID="57013dd4-1ed6-4ebf-a118-c5ea00b8c087" containerName="revoke-keys"
Oct 30 20:59:59.789361 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.789307    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57013dd4-1ed6-4ebf-a118-c5ea00b8c087" containerName="revoke-keys"
Oct 30 20:59:59.789361 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.789329    5071 cpu_manager.go:414] "containerMap: removing container" podUID="7e3d8feb-58a2-40e0-b5b1-94acb06fe606" containerName="istio-proxy"
Oct 30 20:59:59.789361 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.789337    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e3d8feb-58a2-40e0-b5b1-94acb06fe606" containerName="istio-proxy"
Oct 30 20:59:59.789361 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.789353    5071 cpu_manager.go:414] "containerMap: removing container" podUID="7e3d8feb-58a2-40e0-b5b1-94acb06fe606" containerName="istio-proxy"
Oct 30 20:59:59.789482 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.789362    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e3d8feb-58a2-40e0-b5b1-94acb06fe606" containerName="istio-proxy"
Oct 30 20:59:59.789556 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.789542    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="5863cbee-17a0-4b2e-a6e5-c46737b72491" containerName="istio-proxy"
Oct 30 20:59:59.789597 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.789565    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="57013dd4-1ed6-4ebf-a118-c5ea00b8c087" containerName="revoke-keys"
Oct 30 20:59:59.789597 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.789576    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e3d8feb-58a2-40e0-b5b1-94acb06fe606" containerName="istio-proxy"
Oct 30 20:59:59.789597 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.789586    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e3d8feb-58a2-40e0-b5b1-94acb06fe606" containerName="istio-proxy"
Oct 30 20:59:59.793695 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.793675    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-19176d-fwph5"
Oct 30 20:59:59.847224 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.847169    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:59:59.847789 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.847426    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 20:59:59.856181 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.856113    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-19176d-fwph5"]
Oct 30 20:59:59.856542 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.856510    5071 generic.go:356] "Generic (PLEG): container finished" podID="7e3d8feb-58a2-40e0-b5b1-94acb06fe606" containerID="b83be4defe683183511b6df6616f51c17dfcffb8887f8aa3ca17748cb77a761b" exitCode=0
Oct 30 20:59:59.856659 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.856631    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz" event={"ID":"7e3d8feb-58a2-40e0-b5b1-94acb06fe606","Type":"ContainerDied","Data":"b83be4defe683183511b6df6616f51c17dfcffb8887f8aa3ca17748cb77a761b"}
Oct 30 20:59:59.856739 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.856676    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz" event={"ID":"7e3d8feb-58a2-40e0-b5b1-94acb06fe606","Type":"ContainerDied","Data":"197ea65c44c3ebd32982ca242194a03548f7367bdb40fdb781af0aa6a4a3eb00"}
Oct 30 20:59:59.856739 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.856704    5071 scope.go:122] "RemoveContainer" containerID="b83be4defe683183511b6df6616f51c17dfcffb8887f8aa3ca17748cb77a761b"
Oct 30 20:59:59.856922 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.856895    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"
Oct 30 20:59:59.876043 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.876022    5071 scope.go:122] "RemoveContainer" containerID="c79a9635fc675b303fb30614e9496e952b713e415fe8b6b259aa9c279c7f1c2e"
Oct 30 20:59:59.891235 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.890470    5071 scope.go:122] "RemoveContainer" containerID="b83be4defe683183511b6df6616f51c17dfcffb8887f8aa3ca17748cb77a761b"
Oct 30 20:59:59.891562 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:59:59.891532    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b83be4defe683183511b6df6616f51c17dfcffb8887f8aa3ca17748cb77a761b\": container with ID starting with b83be4defe683183511b6df6616f51c17dfcffb8887f8aa3ca17748cb77a761b not found: ID does not exist" containerID="b83be4defe683183511b6df6616f51c17dfcffb8887f8aa3ca17748cb77a761b"
Oct 30 20:59:59.891636 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.891586    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b83be4defe683183511b6df6616f51c17dfcffb8887f8aa3ca17748cb77a761b"} err="failed to get container status \"b83be4defe683183511b6df6616f51c17dfcffb8887f8aa3ca17748cb77a761b\": rpc error: code = NotFound desc = could not find container \"b83be4defe683183511b6df6616f51c17dfcffb8887f8aa3ca17748cb77a761b\": container with ID starting with b83be4defe683183511b6df6616f51c17dfcffb8887f8aa3ca17748cb77a761b not found: ID does not exist"
Oct 30 20:59:59.891636 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.891609    5071 scope.go:122] "RemoveContainer" containerID="c79a9635fc675b303fb30614e9496e952b713e415fe8b6b259aa9c279c7f1c2e"
Oct 30 20:59:59.891996 ip-10-0-139-149 kubenswrapper[5071]: E1030 20:59:59.891975    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c79a9635fc675b303fb30614e9496e952b713e415fe8b6b259aa9c279c7f1c2e\": container with ID starting with c79a9635fc675b303fb30614e9496e952b713e415fe8b6b259aa9c279c7f1c2e not found: ID does not exist" containerID="c79a9635fc675b303fb30614e9496e952b713e415fe8b6b259aa9c279c7f1c2e"
Oct 30 20:59:59.892060 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.892006    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c79a9635fc675b303fb30614e9496e952b713e415fe8b6b259aa9c279c7f1c2e"} err="failed to get container status \"c79a9635fc675b303fb30614e9496e952b713e415fe8b6b259aa9c279c7f1c2e\": rpc error: code = NotFound desc = could not find container \"c79a9635fc675b303fb30614e9496e952b713e415fe8b6b259aa9c279c7f1c2e\": container with ID starting with c79a9635fc675b303fb30614e9496e952b713e415fe8b6b259aa9c279c7f1c2e not found: ID does not exist"
Oct 30 20:59:59.956270 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.956233    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4137c3-e3dc-4a70-b046-75a68d5ae460-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-19176d-fwph5\" (UID: \"ef4137c3-e3dc-4a70-b046-75a68d5ae460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-19176d-fwph5"
Oct 30 20:59:59.956445 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.956396    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6zxk\" (UniqueName: \"kubernetes.io/projected/ef4137c3-e3dc-4a70-b046-75a68d5ae460-kube-api-access-f6zxk\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-19176d-fwph5\" (UID: \"ef4137c3-e3dc-4a70-b046-75a68d5ae460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-19176d-fwph5"
Oct 30 20:59:59.956562 ip-10-0-139-149 kubenswrapper[5071]: I1030 20:59:59.956542    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4137c3-e3dc-4a70-b046-75a68d5ae460-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-19176d-fwph5\" (UID: \"ef4137c3-e3dc-4a70-b046-75a68d5ae460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-19176d-fwph5"
Oct 30 21:00:00.021433 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:00.021402    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"]
Oct 30 21:00:00.028752 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:00.028726    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-9e0fb1-openshift-default-8c75bfccf-tk2xz"]
Oct 30 21:00:00.058211 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:00.058138    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4137c3-e3dc-4a70-b046-75a68d5ae460-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-19176d-fwph5\" (UID: \"ef4137c3-e3dc-4a70-b046-75a68d5ae460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-19176d-fwph5"
Oct 30 21:00:00.058211 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:00.058205    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4137c3-e3dc-4a70-b046-75a68d5ae460-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-19176d-fwph5\" (UID: \"ef4137c3-e3dc-4a70-b046-75a68d5ae460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-19176d-fwph5"
Oct 30 21:00:00.058420 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:00.058299    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6zxk\" (UniqueName: \"kubernetes.io/projected/ef4137c3-e3dc-4a70-b046-75a68d5ae460-kube-api-access-f6zxk\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-19176d-fwph5\" (UID: \"ef4137c3-e3dc-4a70-b046-75a68d5ae460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-19176d-fwph5"
Oct 30 21:00:00.058866 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:00.058845    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4137c3-e3dc-4a70-b046-75a68d5ae460-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-19176d-fwph5\" (UID: \"ef4137c3-e3dc-4a70-b046-75a68d5ae460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-19176d-fwph5"
Oct 30 21:00:00.060885 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:00.060857    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4137c3-e3dc-4a70-b046-75a68d5ae460-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-19176d-fwph5\" (UID: \"ef4137c3-e3dc-4a70-b046-75a68d5ae460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-19176d-fwph5"
Oct 30 21:00:00.116678 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:00.116651    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6zxk\" (UniqueName: \"kubernetes.io/projected/ef4137c3-e3dc-4a70-b046-75a68d5ae460-kube-api-access-f6zxk\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-19176d-fwph5\" (UID: \"ef4137c3-e3dc-4a70-b046-75a68d5ae460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-19176d-fwph5"
Oct 30 21:00:00.168844 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:00.168766    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8kndq"]
Oct 30 21:00:00.174456 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:00.174427    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889900-2gmrr"]
Oct 30 21:00:00.174596 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:00.174559    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8kndq"
Oct 30 21:00:00.178215 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:00.178195    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889900-2gmrr"
Oct 30 21:00:00.201911 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:00.201889    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e3d8feb-58a2-40e0-b5b1-94acb06fe606" path="/var/lib/kubelet/pods/7e3d8feb-58a2-40e0-b5b1-94acb06fe606/volumes"
Oct 30 21:00:00.238782 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:00.238745    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8kndq"]
Oct 30 21:00:00.243758 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:00.243729    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889900-2gmrr"]
Oct 30 21:00:00.260199 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:00.260167    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p7gf\" (UniqueName: \"kubernetes.io/projected/70eb560e-dc33-4be9-a107-a2bed8967828-kube-api-access-7p7gf\") pod \"maas-api-key-cleanup-e2e-worker-w5-386743-29889900-2gmrr\" (UID: \"70eb560e-dc33-4be9-a107-a2bed8967828\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889900-2gmrr"
Oct 30 21:00:00.260347 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:00.260208    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70eb560e-dc33-4be9-a107-a2bed8967828-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-386743-29889900-2gmrr\" (UID: \"70eb560e-dc33-4be9-a107-a2bed8967828\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889900-2gmrr"
Oct 30 21:00:00.260347 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:00.260236    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72be9fd7-0964-4921-85d6-24c9f1609649-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889900-8kndq\" (UID: \"72be9fd7-0964-4921-85d6-24c9f1609649\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8kndq"
Oct 30 21:00:00.260459 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:00.260438    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lljzj\" (UniqueName: \"kubernetes.io/projected/72be9fd7-0964-4921-85d6-24c9f1609649-kube-api-access-lljzj\") pod \"maas-api-key-cleanup-29889900-8kndq\" (UID: \"72be9fd7-0964-4921-85d6-24c9f1609649\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8kndq"
Oct 30 21:00:00.361559 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:00.361481    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lljzj\" (UniqueName: \"kubernetes.io/projected/72be9fd7-0964-4921-85d6-24c9f1609649-kube-api-access-lljzj\") pod \"maas-api-key-cleanup-29889900-8kndq\" (UID: \"72be9fd7-0964-4921-85d6-24c9f1609649\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8kndq"
Oct 30 21:00:00.361559 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:00.361548    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p7gf\" (UniqueName: \"kubernetes.io/projected/70eb560e-dc33-4be9-a107-a2bed8967828-kube-api-access-7p7gf\") pod \"maas-api-key-cleanup-e2e-worker-w5-386743-29889900-2gmrr\" (UID: \"70eb560e-dc33-4be9-a107-a2bed8967828\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889900-2gmrr"
Oct 30 21:00:00.361740 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:00.361571    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70eb560e-dc33-4be9-a107-a2bed8967828-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-386743-29889900-2gmrr\" (UID: \"70eb560e-dc33-4be9-a107-a2bed8967828\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889900-2gmrr"
Oct 30 21:00:00.361740 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:00.361598    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72be9fd7-0964-4921-85d6-24c9f1609649-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889900-8kndq\" (UID: \"72be9fd7-0964-4921-85d6-24c9f1609649\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8kndq"
Oct 30 21:00:00.362260 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:00.362241    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70eb560e-dc33-4be9-a107-a2bed8967828-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-386743-29889900-2gmrr\" (UID: \"70eb560e-dc33-4be9-a107-a2bed8967828\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889900-2gmrr"
Oct 30 21:00:00.362307 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:00.362259    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72be9fd7-0964-4921-85d6-24c9f1609649-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889900-8kndq\" (UID: \"72be9fd7-0964-4921-85d6-24c9f1609649\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8kndq"
Oct 30 21:00:00.407524 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:00.407488    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-19176d-fwph5"
Oct 30 21:00:00.420213 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:00.420189    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lljzj\" (UniqueName: \"kubernetes.io/projected/72be9fd7-0964-4921-85d6-24c9f1609649-kube-api-access-lljzj\") pod \"maas-api-key-cleanup-29889900-8kndq\" (UID: \"72be9fd7-0964-4921-85d6-24c9f1609649\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8kndq"
Oct 30 21:00:00.420353 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:00.420225    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p7gf\" (UniqueName: \"kubernetes.io/projected/70eb560e-dc33-4be9-a107-a2bed8967828-kube-api-access-7p7gf\") pod \"maas-api-key-cleanup-e2e-worker-w5-386743-29889900-2gmrr\" (UID: \"70eb560e-dc33-4be9-a107-a2bed8967828\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889900-2gmrr"
Oct 30 21:00:00.492328 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:00.492296    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8kndq"
Oct 30 21:00:00.500242 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:00.500217    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889900-2gmrr"
Oct 30 21:00:00.782506 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:00.782479    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-19176d-fwph5"]
Oct 30 21:00:00.785027 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:00:00.785000    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef4137c3_e3dc_4a70_b046_75a68d5ae460.slice/crio-fb6bf90fb1cf1f06cb28b467ef91519b941530b13d46c5d404c6e40396360055 WatchSource:0}: Error finding container fb6bf90fb1cf1f06cb28b467ef91519b941530b13d46c5d404c6e40396360055: Status 404 returned error can't find the container with id fb6bf90fb1cf1f06cb28b467ef91519b941530b13d46c5d404c6e40396360055
Oct 30 21:00:00.863767 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:00.863732    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-19176d-fwph5" event={"ID":"ef4137c3-e3dc-4a70-b046-75a68d5ae460","Type":"ContainerStarted","Data":"fb6bf90fb1cf1f06cb28b467ef91519b941530b13d46c5d404c6e40396360055"}
Oct 30 21:00:00.926440 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:00.926412    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8kndq"]
Oct 30 21:00:00.927851 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:00:00.927803    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72be9fd7_0964_4921_85d6_24c9f1609649.slice/crio-b87166018932c35bbc6d44a67d8766d567918e70b8c5d09ec2406dd0f558913d WatchSource:0}: Error finding container b87166018932c35bbc6d44a67d8766d567918e70b8c5d09ec2406dd0f558913d: Status 404 returned error can't find the container with id b87166018932c35bbc6d44a67d8766d567918e70b8c5d09ec2406dd0f558913d
Oct 30 21:00:00.936112 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:00.936084    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889900-2gmrr"]
Oct 30 21:00:01.692026 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:01.691935    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:01.873194 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:01.873154    5071 generic.go:356] "Generic (PLEG): container finished" podID="ef4137c3-e3dc-4a70-b046-75a68d5ae460" containerID="6f0b271a462d319d544e42223abbacf9455cca7a90e421dd824cd09d85501dd6" exitCode=0
Oct 30 21:00:01.873676 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:01.873203    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-19176d-fwph5" event={"ID":"ef4137c3-e3dc-4a70-b046-75a68d5ae460","Type":"ContainerDied","Data":"6f0b271a462d319d544e42223abbacf9455cca7a90e421dd824cd09d85501dd6"}
Oct 30 21:00:01.875005 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:01.874976    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889900-2gmrr" event={"ID":"70eb560e-dc33-4be9-a107-a2bed8967828","Type":"ContainerStarted","Data":"5cb3a1d6863a8b20b51b980a8156d571aac60509b460210560502974a3951718"}
Oct 30 21:00:01.875158 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:01.875018    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889900-2gmrr" event={"ID":"70eb560e-dc33-4be9-a107-a2bed8967828","Type":"ContainerStarted","Data":"ffb14ac23d2ff7e237f2885f9babc7d26bf9ec02cf6c4ddd322caf197cc9f704"}
Oct 30 21:00:01.876463 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:01.876441    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8kndq" event={"ID":"72be9fd7-0964-4921-85d6-24c9f1609649","Type":"ContainerStarted","Data":"c87b55f3b8389be6fca9c7f840bde5a9a0aa3d17919d72075fe534c444517fe7"}
Oct 30 21:00:01.876574 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:01.876483    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8kndq" event={"ID":"72be9fd7-0964-4921-85d6-24c9f1609649","Type":"ContainerStarted","Data":"b87166018932c35bbc6d44a67d8766d567918e70b8c5d09ec2406dd0f558913d"}
Oct 30 21:00:02.109528 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:02.109470    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889900-2gmrr" podStartSLOduration=2.109456826 podStartE2EDuration="2.109456826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:02.106394666 +0000 UTC m=+2014.023416203" watchObservedRunningTime="2026-10-30 21:00:02.109456826 +0000 UTC m=+2014.026478363"
Oct 30 21:00:02.221333 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:02.221226    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8kndq" podStartSLOduration=2.22121233 podStartE2EDuration="2.22121233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:02.218729124 +0000 UTC m=+2014.135750661" watchObservedRunningTime="2026-10-30 21:00:02.22121233 +0000 UTC m=+2014.138233866"
Oct 30 21:00:02.884219 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:02.884181    5071 generic.go:356] "Generic (PLEG): container finished" podID="70eb560e-dc33-4be9-a107-a2bed8967828" containerID="5cb3a1d6863a8b20b51b980a8156d571aac60509b460210560502974a3951718" exitCode=60
Oct 30 21:00:02.884685 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:02.884260    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889900-2gmrr" event={"ID":"70eb560e-dc33-4be9-a107-a2bed8967828","Type":"ContainerDied","Data":"5cb3a1d6863a8b20b51b980a8156d571aac60509b460210560502974a3951718"}
Oct 30 21:00:02.884685 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:02.884545    5071 scope.go:122] "RemoveContainer" containerID="5cb3a1d6863a8b20b51b980a8156d571aac60509b460210560502974a3951718"
Oct 30 21:00:02.885995 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:02.885975    5071 generic.go:356] "Generic (PLEG): container finished" podID="72be9fd7-0964-4921-85d6-24c9f1609649" containerID="c87b55f3b8389be6fca9c7f840bde5a9a0aa3d17919d72075fe534c444517fe7" exitCode=60
Oct 30 21:00:02.886190 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:02.886015    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8kndq" event={"ID":"72be9fd7-0964-4921-85d6-24c9f1609649","Type":"ContainerDied","Data":"c87b55f3b8389be6fca9c7f840bde5a9a0aa3d17919d72075fe534c444517fe7"}
Oct 30 21:00:02.886323 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:02.886300    5071 scope.go:122] "RemoveContainer" containerID="c87b55f3b8389be6fca9c7f840bde5a9a0aa3d17919d72075fe534c444517fe7"
Oct 30 21:00:03.040091 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:03.040068    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-19176d-fwph5"
Oct 30 21:00:03.193169 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:03.193136    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef4137c3-e3dc-4a70-b046-75a68d5ae460-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4137c3-e3dc-4a70-b046-75a68d5ae460-maas-api-openshift-ca\") pod \"ef4137c3-e3dc-4a70-b046-75a68d5ae460\" (UID: \"ef4137c3-e3dc-4a70-b046-75a68d5ae460\") "
Oct 30 21:00:03.193579 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:03.193290    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef4137c3-e3dc-4a70-b046-75a68d5ae460-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4137c3-e3dc-4a70-b046-75a68d5ae460-maas-api-xks-ca\") pod \"ef4137c3-e3dc-4a70-b046-75a68d5ae460\" (UID: \"ef4137c3-e3dc-4a70-b046-75a68d5ae460\") "
Oct 30 21:00:03.193579 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:03.193341    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef4137c3-e3dc-4a70-b046-75a68d5ae460-kube-api-access-f6zxk\" (UniqueName: \"kubernetes.io/projected/ef4137c3-e3dc-4a70-b046-75a68d5ae460-kube-api-access-f6zxk\") pod \"ef4137c3-e3dc-4a70-b046-75a68d5ae460\" (UID: \"ef4137c3-e3dc-4a70-b046-75a68d5ae460\") "
Oct 30 21:00:03.193730 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:03.193628    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef4137c3-e3dc-4a70-b046-75a68d5ae460-maas-api-openshift-ca" pod "ef4137c3-e3dc-4a70-b046-75a68d5ae460" (UID: "ef4137c3-e3dc-4a70-b046-75a68d5ae460"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:03.193798 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:03.193752    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4137c3-e3dc-4a70-b046-75a68d5ae460-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:03.195905 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:03.195876    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef4137c3-e3dc-4a70-b046-75a68d5ae460-maas-api-xks-ca" pod "ef4137c3-e3dc-4a70-b046-75a68d5ae460" (UID: "ef4137c3-e3dc-4a70-b046-75a68d5ae460"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:03.196011 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:03.195903    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef4137c3-e3dc-4a70-b046-75a68d5ae460-kube-api-access-f6zxk" pod "ef4137c3-e3dc-4a70-b046-75a68d5ae460" (UID: "ef4137c3-e3dc-4a70-b046-75a68d5ae460"). InnerVolumeSpecName "kube-api-access-f6zxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:03.294960 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:03.294927    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4137c3-e3dc-4a70-b046-75a68d5ae460-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:03.294960 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:03.294961    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6zxk\" (UniqueName: \"kubernetes.io/projected/ef4137c3-e3dc-4a70-b046-75a68d5ae460-kube-api-access-f6zxk\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:03.894271 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:03.894240    5071 generic.go:356] "Generic (PLEG): container finished" podID="70eb560e-dc33-4be9-a107-a2bed8967828" containerID="882913a52772ed013d551fccd6945d19edea2922a46c091861d1c442c0b8a9cb" exitCode=60
Oct 30 21:00:03.894751 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:03.894326    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889900-2gmrr" event={"ID":"70eb560e-dc33-4be9-a107-a2bed8967828","Type":"ContainerDied","Data":"882913a52772ed013d551fccd6945d19edea2922a46c091861d1c442c0b8a9cb"}
Oct 30 21:00:03.894751 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:03.894395    5071 scope.go:122] "RemoveContainer" containerID="5cb3a1d6863a8b20b51b980a8156d571aac60509b460210560502974a3951718"
Oct 30 21:00:03.894751 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:03.894665    5071 scope.go:122] "RemoveContainer" containerID="882913a52772ed013d551fccd6945d19edea2922a46c091861d1c442c0b8a9cb"
Oct 30 21:00:03.895004 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:00:03.894980    5071 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-386743-29889900-2gmrr_odh-ai-gateway-infra(70eb560e-dc33-4be9-a107-a2bed8967828)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889900-2gmrr" podUID="70eb560e-dc33-4be9-a107-a2bed8967828"
Oct 30 21:00:03.896637 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:03.896616    5071 generic.go:356] "Generic (PLEG): container finished" podID="72be9fd7-0964-4921-85d6-24c9f1609649" containerID="5c0862ceffd0d5aeee49c866e524a5ea612b1fa13f9dc3de0536d3c937bc284e" exitCode=60
Oct 30 21:00:03.896758 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:03.896699    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8kndq" event={"ID":"72be9fd7-0964-4921-85d6-24c9f1609649","Type":"ContainerDied","Data":"5c0862ceffd0d5aeee49c866e524a5ea612b1fa13f9dc3de0536d3c937bc284e"}
Oct 30 21:00:03.896999 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:03.896983    5071 scope.go:122] "RemoveContainer" containerID="5c0862ceffd0d5aeee49c866e524a5ea612b1fa13f9dc3de0536d3c937bc284e"
Oct 30 21:00:03.897284 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:00:03.897258    5071 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889900-8kndq_odh-ai-gateway-infra(72be9fd7-0964-4921-85d6-24c9f1609649)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8kndq" podUID="72be9fd7-0964-4921-85d6-24c9f1609649"
Oct 30 21:00:03.898145 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:03.898126    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-19176d-fwph5" event={"ID":"ef4137c3-e3dc-4a70-b046-75a68d5ae460","Type":"ContainerDied","Data":"fb6bf90fb1cf1f06cb28b467ef91519b941530b13d46c5d404c6e40396360055"}
Oct 30 21:00:03.898256 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:03.898149    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-19176d-fwph5"
Oct 30 21:00:03.898256 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:03.898167    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb6bf90fb1cf1f06cb28b467ef91519b941530b13d46c5d404c6e40396360055"
Oct 30 21:00:03.916628 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:03.916611    5071 scope.go:122] "RemoveContainer" containerID="c87b55f3b8389be6fca9c7f840bde5a9a0aa3d17919d72075fe534c444517fe7"
Oct 30 21:00:04.186432 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.186407    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 21:00:04.307196 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.307166    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c36da32-7ded-4d53-8579-f62445fc61a5-istio-token\" (UniqueName: \"kubernetes.io/projected/2c36da32-7ded-4d53-8579-f62445fc61a5-istio-token\") pod \"2c36da32-7ded-4d53-8579-f62445fc61a5\" (UID: \"2c36da32-7ded-4d53-8579-f62445fc61a5\") "
Oct 30 21:00:04.307633 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.307218    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c36da32-7ded-4d53-8579-f62445fc61a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c36da32-7ded-4d53-8579-f62445fc61a5-istio-podinfo\") pod \"2c36da32-7ded-4d53-8579-f62445fc61a5\" (UID: \"2c36da32-7ded-4d53-8579-f62445fc61a5\") "
Oct 30 21:00:04.307633 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.307392    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c36da32-7ded-4d53-8579-f62445fc61a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c36da32-7ded-4d53-8579-f62445fc61a5-workload-certs\") pod \"2c36da32-7ded-4d53-8579-f62445fc61a5\" (UID: \"2c36da32-7ded-4d53-8579-f62445fc61a5\") "
Oct 30 21:00:04.307633 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.307511    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c36da32-7ded-4d53-8579-f62445fc61a5-kube-api-access-k6h9l\" (UniqueName: \"kubernetes.io/projected/2c36da32-7ded-4d53-8579-f62445fc61a5-kube-api-access-k6h9l\") pod \"2c36da32-7ded-4d53-8579-f62445fc61a5\" (UID: \"2c36da32-7ded-4d53-8579-f62445fc61a5\") "
Oct 30 21:00:04.307633 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.307547    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c36da32-7ded-4d53-8579-f62445fc61a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c36da32-7ded-4d53-8579-f62445fc61a5-workload-socket\") pod \"2c36da32-7ded-4d53-8579-f62445fc61a5\" (UID: \"2c36da32-7ded-4d53-8579-f62445fc61a5\") "
Oct 30 21:00:04.307633 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.307579    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c36da32-7ded-4d53-8579-f62445fc61a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c36da32-7ded-4d53-8579-f62445fc61a5-istiod-ca-cert\") pod \"2c36da32-7ded-4d53-8579-f62445fc61a5\" (UID: \"2c36da32-7ded-4d53-8579-f62445fc61a5\") "
Oct 30 21:00:04.308094 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.307650    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c36da32-7ded-4d53-8579-f62445fc61a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c36da32-7ded-4d53-8579-f62445fc61a5-istio-data\") pod \"2c36da32-7ded-4d53-8579-f62445fc61a5\" (UID: \"2c36da32-7ded-4d53-8579-f62445fc61a5\") "
Oct 30 21:00:04.308094 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.307784    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c36da32-7ded-4d53-8579-f62445fc61a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c36da32-7ded-4d53-8579-f62445fc61a5-credential-socket\") pod \"2c36da32-7ded-4d53-8579-f62445fc61a5\" (UID: \"2c36da32-7ded-4d53-8579-f62445fc61a5\") "
Oct 30 21:00:04.308094 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.307883    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c36da32-7ded-4d53-8579-f62445fc61a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c36da32-7ded-4d53-8579-f62445fc61a5-istio-envoy\") pod \"2c36da32-7ded-4d53-8579-f62445fc61a5\" (UID: \"2c36da32-7ded-4d53-8579-f62445fc61a5\") "
Oct 30 21:00:04.308094 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.308050    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c36da32-7ded-4d53-8579-f62445fc61a5-workload-socket" pod "2c36da32-7ded-4d53-8579-f62445fc61a5" (UID: "2c36da32-7ded-4d53-8579-f62445fc61a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:04.308352 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.308319    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c36da32-7ded-4d53-8579-f62445fc61a5-credential-socket" pod "2c36da32-7ded-4d53-8579-f62445fc61a5" (UID: "2c36da32-7ded-4d53-8579-f62445fc61a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:04.308629 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.308562    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c36da32-7ded-4d53-8579-f62445fc61a5-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:04.308629 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.308588    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c36da32-7ded-4d53-8579-f62445fc61a5-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:04.310624 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.310598    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c36da32-7ded-4d53-8579-f62445fc61a5-istio-podinfo" pod "2c36da32-7ded-4d53-8579-f62445fc61a5" (UID: "2c36da32-7ded-4d53-8579-f62445fc61a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:00:04.310903 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.310879    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c36da32-7ded-4d53-8579-f62445fc61a5-kube-api-access-k6h9l" pod "2c36da32-7ded-4d53-8579-f62445fc61a5" (UID: "2c36da32-7ded-4d53-8579-f62445fc61a5"). InnerVolumeSpecName "kube-api-access-k6h9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:04.311240 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.311220    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c36da32-7ded-4d53-8579-f62445fc61a5-istio-token" pod "2c36da32-7ded-4d53-8579-f62445fc61a5" (UID: "2c36da32-7ded-4d53-8579-f62445fc61a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:04.312472 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.312419    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c36da32-7ded-4d53-8579-f62445fc61a5-workload-certs" pod "2c36da32-7ded-4d53-8579-f62445fc61a5" (UID: "2c36da32-7ded-4d53-8579-f62445fc61a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:04.312594 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.312480    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c36da32-7ded-4d53-8579-f62445fc61a5-istio-envoy" pod "2c36da32-7ded-4d53-8579-f62445fc61a5" (UID: "2c36da32-7ded-4d53-8579-f62445fc61a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:04.312667 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.312609    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c36da32-7ded-4d53-8579-f62445fc61a5-istio-data" pod "2c36da32-7ded-4d53-8579-f62445fc61a5" (UID: "2c36da32-7ded-4d53-8579-f62445fc61a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:04.312729 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.312657    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c36da32-7ded-4d53-8579-f62445fc61a5-istiod-ca-cert" pod "2c36da32-7ded-4d53-8579-f62445fc61a5" (UID: "2c36da32-7ded-4d53-8579-f62445fc61a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:04.409406 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.409370    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c36da32-7ded-4d53-8579-f62445fc61a5-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:04.409406 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.409402    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c36da32-7ded-4d53-8579-f62445fc61a5-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:04.409406 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.409412    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c36da32-7ded-4d53-8579-f62445fc61a5-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:04.409628 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.409420    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c36da32-7ded-4d53-8579-f62445fc61a5-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:04.409628 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.409430    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6h9l\" (UniqueName: \"kubernetes.io/projected/2c36da32-7ded-4d53-8579-f62445fc61a5-kube-api-access-k6h9l\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:04.409628 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.409440    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c36da32-7ded-4d53-8579-f62445fc61a5-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:04.409628 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.409448    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c36da32-7ded-4d53-8579-f62445fc61a5-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:04.915428 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.915384    5071 scope.go:122] "RemoveContainer" containerID="5c0862ceffd0d5aeee49c866e524a5ea612b1fa13f9dc3de0536d3c937bc284e"
Oct 30 21:00:04.915945 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:00:04.915648    5071 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889900-8kndq_odh-ai-gateway-infra(72be9fd7-0964-4921-85d6-24c9f1609649)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8kndq" podUID="72be9fd7-0964-4921-85d6-24c9f1609649"
Oct 30 21:00:04.917284 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.917255    5071 generic.go:356] "Generic (PLEG): container finished" podID="2c36da32-7ded-4d53-8579-f62445fc61a5" containerID="15cdf37757674bf07863987b38358fb28ad8564879f10999a9c3ca279dcdf069" exitCode=0
Oct 30 21:00:04.917496 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.917323    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"
Oct 30 21:00:04.917496 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.917352    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw" event={"ID":"2c36da32-7ded-4d53-8579-f62445fc61a5","Type":"ContainerDied","Data":"15cdf37757674bf07863987b38358fb28ad8564879f10999a9c3ca279dcdf069"}
Oct 30 21:00:04.917496 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.917431    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw" event={"ID":"2c36da32-7ded-4d53-8579-f62445fc61a5","Type":"ContainerDied","Data":"f6375e1cdeec890e8b4375d84d50563eb8589a9fa132c7c92263f6855f961bce"}
Oct 30 21:00:04.917496 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.917460    5071 scope.go:122] "RemoveContainer" containerID="15cdf37757674bf07863987b38358fb28ad8564879f10999a9c3ca279dcdf069"
Oct 30 21:00:04.919407 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.919389    5071 scope.go:122] "RemoveContainer" containerID="882913a52772ed013d551fccd6945d19edea2922a46c091861d1c442c0b8a9cb"
Oct 30 21:00:04.919645 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:00:04.919622    5071 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-386743-29889900-2gmrr_odh-ai-gateway-infra(70eb560e-dc33-4be9-a107-a2bed8967828)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889900-2gmrr" podUID="70eb560e-dc33-4be9-a107-a2bed8967828"
Oct 30 21:00:04.928524 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.928509    5071 scope.go:122] "RemoveContainer" containerID="7c11413bb0d8e2fdbfdd114869b7450d0a54292162df6de3363659ca73b68163"
Oct 30 21:00:04.937218 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.937203    5071 scope.go:122] "RemoveContainer" containerID="15cdf37757674bf07863987b38358fb28ad8564879f10999a9c3ca279dcdf069"
Oct 30 21:00:04.937465 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:00:04.937446    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15cdf37757674bf07863987b38358fb28ad8564879f10999a9c3ca279dcdf069\": container with ID starting with 15cdf37757674bf07863987b38358fb28ad8564879f10999a9c3ca279dcdf069 not found: ID does not exist" containerID="15cdf37757674bf07863987b38358fb28ad8564879f10999a9c3ca279dcdf069"
Oct 30 21:00:04.937510 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.937472    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15cdf37757674bf07863987b38358fb28ad8564879f10999a9c3ca279dcdf069"} err="failed to get container status \"15cdf37757674bf07863987b38358fb28ad8564879f10999a9c3ca279dcdf069\": rpc error: code = NotFound desc = could not find container \"15cdf37757674bf07863987b38358fb28ad8564879f10999a9c3ca279dcdf069\": container with ID starting with 15cdf37757674bf07863987b38358fb28ad8564879f10999a9c3ca279dcdf069 not found: ID does not exist"
Oct 30 21:00:04.937510 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.937490    5071 scope.go:122] "RemoveContainer" containerID="7c11413bb0d8e2fdbfdd114869b7450d0a54292162df6de3363659ca73b68163"
Oct 30 21:00:04.937721 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:00:04.937706    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c11413bb0d8e2fdbfdd114869b7450d0a54292162df6de3363659ca73b68163\": container with ID starting with 7c11413bb0d8e2fdbfdd114869b7450d0a54292162df6de3363659ca73b68163 not found: ID does not exist" containerID="7c11413bb0d8e2fdbfdd114869b7450d0a54292162df6de3363659ca73b68163"
Oct 30 21:00:04.937755 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:04.937727    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c11413bb0d8e2fdbfdd114869b7450d0a54292162df6de3363659ca73b68163"} err="failed to get container status \"7c11413bb0d8e2fdbfdd114869b7450d0a54292162df6de3363659ca73b68163\": rpc error: code = NotFound desc = could not find container \"7c11413bb0d8e2fdbfdd114869b7450d0a54292162df6de3363659ca73b68163\": container with ID starting with 7c11413bb0d8e2fdbfdd114869b7450d0a54292162df6de3363659ca73b68163 not found: ID does not exist"
Oct 30 21:00:05.267253 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:05.267168    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"]
Oct 30 21:00:05.273032 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:05.272996    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d9b3f6-openshift-default-7d877dc56c-n4jdw"]
Oct 30 21:00:06.203045 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:06.203007    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c36da32-7ded-4d53-8579-f62445fc61a5" path="/var/lib/kubelet/pods/2c36da32-7ded-4d53-8579-f62445fc61a5/volumes"
Oct 30 21:00:06.223756 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:06.223721    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:06.284937 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:06.284898    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889900-2gmrr"]
Oct 30 21:00:06.465669 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:06.465607    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889900-2gmrr"
Oct 30 21:00:06.594053 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:06.594016    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-ee3fad-67dffbb55d-vjzhn"]
Oct 30 21:00:06.635983 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:06.635953    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70eb560e-dc33-4be9-a107-a2bed8967828-kube-api-access-7p7gf\" (UniqueName: \"kubernetes.io/projected/70eb560e-dc33-4be9-a107-a2bed8967828-kube-api-access-7p7gf\") pod \"70eb560e-dc33-4be9-a107-a2bed8967828\" (UID: \"70eb560e-dc33-4be9-a107-a2bed8967828\") "
Oct 30 21:00:06.636152 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:06.636029    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70eb560e-dc33-4be9-a107-a2bed8967828-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70eb560e-dc33-4be9-a107-a2bed8967828-maas-api-service-ca\") pod \"70eb560e-dc33-4be9-a107-a2bed8967828\" (UID: \"70eb560e-dc33-4be9-a107-a2bed8967828\") "
Oct 30 21:00:06.636366 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:06.636346    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70eb560e-dc33-4be9-a107-a2bed8967828-maas-api-service-ca" pod "70eb560e-dc33-4be9-a107-a2bed8967828" (UID: "70eb560e-dc33-4be9-a107-a2bed8967828"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:06.638264 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:06.638234    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70eb560e-dc33-4be9-a107-a2bed8967828-kube-api-access-7p7gf" pod "70eb560e-dc33-4be9-a107-a2bed8967828" (UID: "70eb560e-dc33-4be9-a107-a2bed8967828"). InnerVolumeSpecName "kube-api-access-7p7gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:06.697448 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:06.697413    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-ee3fad-69f6cc779f-plr9b"]
Oct 30 21:00:06.737563 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:06.737480    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p7gf\" (UniqueName: \"kubernetes.io/projected/70eb560e-dc33-4be9-a107-a2bed8967828-kube-api-access-7p7gf\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:06.737563 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:06.737515    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70eb560e-dc33-4be9-a107-a2bed8967828-maas-api-service-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:06.935708 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:06.935676    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889900-2gmrr" event={"ID":"70eb560e-dc33-4be9-a107-a2bed8967828","Type":"ContainerDied","Data":"ffb14ac23d2ff7e237f2885f9babc7d26bf9ec02cf6c4ddd322caf197cc9f704"}
Oct 30 21:00:06.935895 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:06.935716    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889900-2gmrr"
Oct 30 21:00:06.935895 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:06.935735    5071 scope.go:122] "RemoveContainer" containerID="882913a52772ed013d551fccd6945d19edea2922a46c091861d1c442c0b8a9cb"
Oct 30 21:00:06.980853 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:06.980802    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:07.101971 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:07.101935    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889900-2gmrr"]
Oct 30 21:00:07.107349 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:07.107325    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-386743-29889900-2gmrr"]
Oct 30 21:00:07.145968 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:07.145694    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:07.392900 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:07.392810    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7fb8880-jxtnj"]
Oct 30 21:00:07.393315 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:07.393301    5071 cpu_manager.go:414] "containerMap: removing container" podUID="70eb560e-dc33-4be9-a107-a2bed8967828" containerName="cleanup"
Oct 30 21:00:07.393363 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:07.393317    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70eb560e-dc33-4be9-a107-a2bed8967828" containerName="cleanup"
Oct 30 21:00:07.393363 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:07.393345    5071 cpu_manager.go:414] "containerMap: removing container" podUID="70eb560e-dc33-4be9-a107-a2bed8967828" containerName="cleanup"
Oct 30 21:00:07.393363 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:07.393350    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70eb560e-dc33-4be9-a107-a2bed8967828" containerName="cleanup"
Oct 30 21:00:07.393456 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:07.393366    5071 cpu_manager.go:414] "containerMap: removing container" podUID="ef4137c3-e3dc-4a70-b046-75a68d5ae460" containerName="revoke-keys"
Oct 30 21:00:07.393456 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:07.393372    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef4137c3-e3dc-4a70-b046-75a68d5ae460" containerName="revoke-keys"
Oct 30 21:00:07.393456 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:07.393387    5071 cpu_manager.go:414] "containerMap: removing container" podUID="2c36da32-7ded-4d53-8579-f62445fc61a5" containerName="istio-proxy"
Oct 30 21:00:07.393456 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:07.393392    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c36da32-7ded-4d53-8579-f62445fc61a5" containerName="istio-proxy"
Oct 30 21:00:07.393456 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:07.393397    5071 cpu_manager.go:414] "containerMap: removing container" podUID="2c36da32-7ded-4d53-8579-f62445fc61a5" containerName="istio-proxy"
Oct 30 21:00:07.393456 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:07.393402    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c36da32-7ded-4d53-8579-f62445fc61a5" containerName="istio-proxy"
Oct 30 21:00:07.393639 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:07.393467    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef4137c3-e3dc-4a70-b046-75a68d5ae460" containerName="revoke-keys"
Oct 30 21:00:07.393639 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:07.393477    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c36da32-7ded-4d53-8579-f62445fc61a5" containerName="istio-proxy"
Oct 30 21:00:07.393639 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:07.393482    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c36da32-7ded-4d53-8579-f62445fc61a5" containerName="istio-proxy"
Oct 30 21:00:07.393639 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:07.393489    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="70eb560e-dc33-4be9-a107-a2bed8967828" containerName="cleanup"
Oct 30 21:00:07.398097 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:07.398079    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7fb8880-jxtnj"
Oct 30 21:00:07.451803 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:07.451777    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:00:07.454762 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:07.454740    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7fb8880-jxtnj"]
Oct 30 21:00:07.547008 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:07.546967    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc29059-37fb-4f25-8809-148d6115a8a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7fb8880-jxtnj\" (UID: \"6dc29059-37fb-4f25-8809-148d6115a8a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7fb8880-jxtnj"
Oct 30 21:00:07.547008 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:07.547008    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc29059-37fb-4f25-8809-148d6115a8a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7fb8880-jxtnj\" (UID: \"6dc29059-37fb-4f25-8809-148d6115a8a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7fb8880-jxtnj"
Oct 30 21:00:07.547265 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:07.547200    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfqg4\" (UniqueName: \"kubernetes.io/projected/6dc29059-37fb-4f25-8809-148d6115a8a5-kube-api-access-qfqg4\") pod \"maas-api-revoke-sub-e7fb8880-jxtnj\" (UID: \"6dc29059-37fb-4f25-8809-148d6115a8a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7fb8880-jxtnj"
Oct 30 21:00:07.648322 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:07.648228    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc29059-37fb-4f25-8809-148d6115a8a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7fb8880-jxtnj\" (UID: \"6dc29059-37fb-4f25-8809-148d6115a8a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7fb8880-jxtnj"
Oct 30 21:00:07.648322 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:07.648284    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc29059-37fb-4f25-8809-148d6115a8a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7fb8880-jxtnj\" (UID: \"6dc29059-37fb-4f25-8809-148d6115a8a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7fb8880-jxtnj"
Oct 30 21:00:07.648565 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:07.648439    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfqg4\" (UniqueName: \"kubernetes.io/projected/6dc29059-37fb-4f25-8809-148d6115a8a5-kube-api-access-qfqg4\") pod \"maas-api-revoke-sub-e7fb8880-jxtnj\" (UID: \"6dc29059-37fb-4f25-8809-148d6115a8a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7fb8880-jxtnj"
Oct 30 21:00:07.649089 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:07.649065    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc29059-37fb-4f25-8809-148d6115a8a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7fb8880-jxtnj\" (UID: \"6dc29059-37fb-4f25-8809-148d6115a8a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7fb8880-jxtnj"
Oct 30 21:00:07.650880 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:07.650857    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc29059-37fb-4f25-8809-148d6115a8a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7fb8880-jxtnj\" (UID: \"6dc29059-37fb-4f25-8809-148d6115a8a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7fb8880-jxtnj"
Oct 30 21:00:07.706620 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:07.706581    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfqg4\" (UniqueName: \"kubernetes.io/projected/6dc29059-37fb-4f25-8809-148d6115a8a5-kube-api-access-qfqg4\") pod \"maas-api-revoke-sub-e7fb8880-jxtnj\" (UID: \"6dc29059-37fb-4f25-8809-148d6115a8a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7fb8880-jxtnj"
Oct 30 21:00:07.711274 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:07.711253    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7fb8880-jxtnj"
Oct 30 21:00:07.783671 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:07.783636    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:08.069637 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:08.069601    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:08.087500 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:08.087474    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7fb8880-jxtnj"]
Oct 30 21:00:08.090604 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:00:08.090551    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc29059_37fb_4f25_8809_148d6115a8a5.slice/crio-a24358f32fd92519aa995e68715b370211cadc849e29d800d653e6edf9e6a5ab WatchSource:0}: Error finding container a24358f32fd92519aa995e68715b370211cadc849e29d800d653e6edf9e6a5ab: Status 404 returned error can't find the container with id a24358f32fd92519aa995e68715b370211cadc849e29d800d653e6edf9e6a5ab
Oct 30 21:00:08.203132 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:08.203099    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70eb560e-dc33-4be9-a107-a2bed8967828" path="/var/lib/kubelet/pods/70eb560e-dc33-4be9-a107-a2bed8967828/volumes"
Oct 30 21:00:08.266268 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:08.266236    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:08.823113 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:08.823074    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:08.957918 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:08.957878    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7fb8880-jxtnj" event={"ID":"6dc29059-37fb-4f25-8809-148d6115a8a5","Type":"ContainerStarted","Data":"b2128feb84214f88ac6416425ea62b831fb75ab5e78e01d8230e9282b163297a"}
Oct 30 21:00:08.958243 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:08.958220    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7fb8880-jxtnj" event={"ID":"6dc29059-37fb-4f25-8809-148d6115a8a5","Type":"ContainerStarted","Data":"a24358f32fd92519aa995e68715b370211cadc849e29d800d653e6edf9e6a5ab"}
Oct 30 21:00:09.075768 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:09.075680    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7fb8880-jxtnj" podStartSLOduration=2.075668091 podStartE2EDuration="2.075668091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:09.075533189 +0000 UTC m=+2020.992554724" watchObservedRunningTime="2026-10-30 21:00:09.075668091 +0000 UTC m=+2020.992689679"
Oct 30 21:00:09.965149 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:09.965111    5071 generic.go:356] "Generic (PLEG): container finished" podID="6dc29059-37fb-4f25-8809-148d6115a8a5" containerID="b2128feb84214f88ac6416425ea62b831fb75ab5e78e01d8230e9282b163297a" exitCode=0
Oct 30 21:00:09.965649 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:09.965191    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7fb8880-jxtnj" event={"ID":"6dc29059-37fb-4f25-8809-148d6115a8a5","Type":"ContainerDied","Data":"b2128feb84214f88ac6416425ea62b831fb75ab5e78e01d8230e9282b163297a"}
Oct 30 21:00:11.123688 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:11.123655    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7fb8880-jxtnj"
Oct 30 21:00:11.286560 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:11.286477    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dc29059-37fb-4f25-8809-148d6115a8a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc29059-37fb-4f25-8809-148d6115a8a5-maas-api-xks-ca\") pod \"6dc29059-37fb-4f25-8809-148d6115a8a5\" (UID: \"6dc29059-37fb-4f25-8809-148d6115a8a5\") "
Oct 30 21:00:11.286560 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:11.286536    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dc29059-37fb-4f25-8809-148d6115a8a5-kube-api-access-qfqg4\" (UniqueName: \"kubernetes.io/projected/6dc29059-37fb-4f25-8809-148d6115a8a5-kube-api-access-qfqg4\") pod \"6dc29059-37fb-4f25-8809-148d6115a8a5\" (UID: \"6dc29059-37fb-4f25-8809-148d6115a8a5\") "
Oct 30 21:00:11.286810 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:11.286564    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dc29059-37fb-4f25-8809-148d6115a8a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc29059-37fb-4f25-8809-148d6115a8a5-maas-api-openshift-ca\") pod \"6dc29059-37fb-4f25-8809-148d6115a8a5\" (UID: \"6dc29059-37fb-4f25-8809-148d6115a8a5\") "
Oct 30 21:00:11.287026 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:11.286999    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dc29059-37fb-4f25-8809-148d6115a8a5-maas-api-openshift-ca" pod "6dc29059-37fb-4f25-8809-148d6115a8a5" (UID: "6dc29059-37fb-4f25-8809-148d6115a8a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:11.289034 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:11.289009    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dc29059-37fb-4f25-8809-148d6115a8a5-maas-api-xks-ca" pod "6dc29059-37fb-4f25-8809-148d6115a8a5" (UID: "6dc29059-37fb-4f25-8809-148d6115a8a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:11.289157 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:11.289123    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dc29059-37fb-4f25-8809-148d6115a8a5-kube-api-access-qfqg4" pod "6dc29059-37fb-4f25-8809-148d6115a8a5" (UID: "6dc29059-37fb-4f25-8809-148d6115a8a5"). InnerVolumeSpecName "kube-api-access-qfqg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:11.316400 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:11.316375    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-19176d-ff6f9d87d-5l9qg"]
Oct 30 21:00:11.388578 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:11.388548    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc29059-37fb-4f25-8809-148d6115a8a5-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:11.388578 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:11.388572    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfqg4\" (UniqueName: \"kubernetes.io/projected/6dc29059-37fb-4f25-8809-148d6115a8a5-kube-api-access-qfqg4\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:11.388578 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:11.388584    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc29059-37fb-4f25-8809-148d6115a8a5-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:11.436012 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:11.435982    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-19176d-69f48c59bc-wgtb8"]
Oct 30 21:00:11.595253 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:11.595216    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-ee3fad-67dffbb55d-vjzhn" podUID="20a5960c-1432-48a8-8b82-730d6973647d" containerName="payload-processing" containerID="cri-o://2dd06967f87a7f0bb535f1b3b6f992a79f9cf637fde6bbe8d1089c6cb4bafe1e" gracePeriod=25
Oct 30 21:00:11.698450 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:11.698380    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee3fad-69f6cc779f-plr9b" podUID="56a15d9e-6de3-47ab-8045-4a1c3618bc6a" containerName="payload-pre-processing" containerID="cri-o://14f626459dd48264a397c532e3c63cdf0dcbf06db5bcf38fd73177aa2b9eff50" gracePeriod=25
Oct 30 21:00:11.717105 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:11.717071    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:11.923810 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:11.923715    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:11.980595 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:11.980559    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7fb8880-jxtnj" event={"ID":"6dc29059-37fb-4f25-8809-148d6115a8a5","Type":"ContainerDied","Data":"a24358f32fd92519aa995e68715b370211cadc849e29d800d653e6edf9e6a5ab"}
Oct 30 21:00:11.980812 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:11.980616    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a24358f32fd92519aa995e68715b370211cadc849e29d800d653e6edf9e6a5ab"
Oct 30 21:00:11.980812 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:11.980565    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7fb8880-jxtnj"
Oct 30 21:00:13.594838 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:13.594796    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:14.238815 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:14.238782    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:00:16.093221 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:16.093180    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:16.194041 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:16.194005    5071 scope.go:122] "RemoveContainer" containerID="5c0862ceffd0d5aeee49c866e524a5ea612b1fa13f9dc3de0536d3c937bc284e"
Oct 30 21:00:16.260324 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:16.260282    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:16.317446 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:16.317410    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-19176d-ff6f9d87d-5l9qg" podUID="b6a43db1-079d-4a4e-b048-641193df5ea4" containerName="payload-processing" containerID="cri-o://06d27e6cef247833ec88b653239a0f341dcaa36127b972ee4c377c9c42123d02" gracePeriod=25
Oct 30 21:00:16.408939 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:16.408853    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"]
Oct 30 21:00:16.409163 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:16.409136    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9" podUID="c80fd141-e109-4216-a3f1-39d326d72ab1" containerName="istio-proxy" containerID="cri-o://5d588107127690f85d2e4027d3b78fcf6e4b670f2f33679c83e80fbcdced4d8b" gracePeriod=30
Oct 30 21:00:16.436754 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:16.436722    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-19176d-69f48c59bc-wgtb8" podUID="9104e5d7-09e8-4622-b783-69506c636052" containerName="payload-pre-processing" containerID="cri-o://77d8bf64718549233ad972f0369d7b4b6ac5b048e6e5fd33f60180b0ed2b245e" gracePeriod=25
Oct 30 21:00:16.439433 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:16.439283    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:16.644641 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:16.644608    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-19176d-ff6f9d87d-5l9qg"
Oct 30 21:00:16.706466 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:16.706438    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-19176d-69f48c59bc-wgtb8"
Oct 30 21:00:16.727918 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:00:16.727884    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:00:16.756062 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:16.756027    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9104e5d7-09e8-4622-b783-69506c636052-kube-api-access-cbvsv\" (UniqueName: \"kubernetes.io/projected/9104e5d7-09e8-4622-b783-69506c636052-kube-api-access-cbvsv\") pod \"9104e5d7-09e8-4622-b783-69506c636052\" (UID: \"9104e5d7-09e8-4622-b783-69506c636052\") "
Oct 30 21:00:16.756235 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:16.756152    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6a43db1-079d-4a4e-b048-641193df5ea4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6a43db1-079d-4a4e-b048-641193df5ea4-plugins-config-volume\") pod \"b6a43db1-079d-4a4e-b048-641193df5ea4\" (UID: \"b6a43db1-079d-4a4e-b048-641193df5ea4\") "
Oct 30 21:00:16.756235 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:16.756171    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9104e5d7-09e8-4622-b783-69506c636052-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9104e5d7-09e8-4622-b783-69506c636052-plugins-config-volume\") pod \"9104e5d7-09e8-4622-b783-69506c636052\" (UID: \"9104e5d7-09e8-4622-b783-69506c636052\") "
Oct 30 21:00:16.756235 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:16.756188    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6a43db1-079d-4a4e-b048-641193df5ea4-kube-api-access-2lsr8\" (UniqueName: \"kubernetes.io/projected/b6a43db1-079d-4a4e-b048-641193df5ea4-kube-api-access-2lsr8\") pod \"b6a43db1-079d-4a4e-b048-641193df5ea4\" (UID: \"b6a43db1-079d-4a4e-b048-641193df5ea4\") "
Oct 30 21:00:16.756569 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:16.756545    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6a43db1-079d-4a4e-b048-641193df5ea4-plugins-config-volume" pod "b6a43db1-079d-4a4e-b048-641193df5ea4" (UID: "b6a43db1-079d-4a4e-b048-641193df5ea4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:16.756634 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:16.756562    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9104e5d7-09e8-4622-b783-69506c636052-plugins-config-volume" pod "9104e5d7-09e8-4622-b783-69506c636052" (UID: "9104e5d7-09e8-4622-b783-69506c636052"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:16.758255 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:16.758235    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9104e5d7-09e8-4622-b783-69506c636052-kube-api-access-cbvsv" pod "9104e5d7-09e8-4622-b783-69506c636052" (UID: "9104e5d7-09e8-4622-b783-69506c636052"). InnerVolumeSpecName "kube-api-access-cbvsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:16.758433 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:16.758413    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6a43db1-079d-4a4e-b048-641193df5ea4-kube-api-access-2lsr8" pod "b6a43db1-079d-4a4e-b048-641193df5ea4" (UID: "b6a43db1-079d-4a4e-b048-641193df5ea4"). InnerVolumeSpecName "kube-api-access-2lsr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:16.857779 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:16.857745    5071 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6a43db1-079d-4a4e-b048-641193df5ea4-plugins-config-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:16.857779 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:16.857774    5071 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9104e5d7-09e8-4622-b783-69506c636052-plugins-config-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:16.857779 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:16.857785    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lsr8\" (UniqueName: \"kubernetes.io/projected/b6a43db1-079d-4a4e-b048-641193df5ea4-kube-api-access-2lsr8\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:16.858049 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:16.857794    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbvsv\" (UniqueName: \"kubernetes.io/projected/9104e5d7-09e8-4622-b783-69506c636052-kube-api-access-cbvsv\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:17.060395 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:17.060359    5071 generic.go:356] "Generic (PLEG): container finished" podID="b6a43db1-079d-4a4e-b048-641193df5ea4" containerID="06d27e6cef247833ec88b653239a0f341dcaa36127b972ee4c377c9c42123d02" exitCode=0
Oct 30 21:00:17.060561 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:17.060407    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-19176d-ff6f9d87d-5l9qg"
Oct 30 21:00:17.060561 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:17.060444    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-19176d-ff6f9d87d-5l9qg" event={"ID":"b6a43db1-079d-4a4e-b048-641193df5ea4","Type":"ContainerDied","Data":"06d27e6cef247833ec88b653239a0f341dcaa36127b972ee4c377c9c42123d02"}
Oct 30 21:00:17.060561 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:17.060497    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-19176d-ff6f9d87d-5l9qg" event={"ID":"b6a43db1-079d-4a4e-b048-641193df5ea4","Type":"ContainerDied","Data":"3e27fadb79bf6a1eb712fc84bcaac46cd81541ff4e7d77b4222d9457a4dbc610"}
Oct 30 21:00:17.060561 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:17.060519    5071 scope.go:122] "RemoveContainer" containerID="06d27e6cef247833ec88b653239a0f341dcaa36127b972ee4c377c9c42123d02"
Oct 30 21:00:17.062807 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:17.062782    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8kndq" event={"ID":"72be9fd7-0964-4921-85d6-24c9f1609649","Type":"ContainerStarted","Data":"f39ad4cbf4c6975fd8e91a058c98549266fc5025f055b87e873e81af918e3768"}
Oct 30 21:00:17.064320 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:17.064294    5071 generic.go:356] "Generic (PLEG): container finished" podID="9104e5d7-09e8-4622-b783-69506c636052" containerID="77d8bf64718549233ad972f0369d7b4b6ac5b048e6e5fd33f60180b0ed2b245e" exitCode=0
Oct 30 21:00:17.064418 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:17.064332    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-19176d-69f48c59bc-wgtb8"
Oct 30 21:00:17.064418 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:17.064384    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-19176d-69f48c59bc-wgtb8" event={"ID":"9104e5d7-09e8-4622-b783-69506c636052","Type":"ContainerDied","Data":"77d8bf64718549233ad972f0369d7b4b6ac5b048e6e5fd33f60180b0ed2b245e"}
Oct 30 21:00:17.064514 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:17.064441    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-19176d-69f48c59bc-wgtb8" event={"ID":"9104e5d7-09e8-4622-b783-69506c636052","Type":"ContainerDied","Data":"a2b16fba6163ba76353cb884a654b0b4bf5b8c7f896143edac39e222531f44ca"}
Oct 30 21:00:17.072452 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:17.072428    5071 scope.go:122] "RemoveContainer" containerID="06d27e6cef247833ec88b653239a0f341dcaa36127b972ee4c377c9c42123d02"
Oct 30 21:00:17.072763 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:00:17.072741    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06d27e6cef247833ec88b653239a0f341dcaa36127b972ee4c377c9c42123d02\": container with ID starting with 06d27e6cef247833ec88b653239a0f341dcaa36127b972ee4c377c9c42123d02 not found: ID does not exist" containerID="06d27e6cef247833ec88b653239a0f341dcaa36127b972ee4c377c9c42123d02"
Oct 30 21:00:17.072883 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:17.072778    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06d27e6cef247833ec88b653239a0f341dcaa36127b972ee4c377c9c42123d02"} err="failed to get container status \"06d27e6cef247833ec88b653239a0f341dcaa36127b972ee4c377c9c42123d02\": rpc error: code = NotFound desc = could not find container \"06d27e6cef247833ec88b653239a0f341dcaa36127b972ee4c377c9c42123d02\": container with ID starting with 06d27e6cef247833ec88b653239a0f341dcaa36127b972ee4c377c9c42123d02 not found: ID does not exist"
Oct 30 21:00:17.072883 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:17.072804    5071 scope.go:122] "RemoveContainer" containerID="77d8bf64718549233ad972f0369d7b4b6ac5b048e6e5fd33f60180b0ed2b245e"
Oct 30 21:00:17.081699 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:17.081668    5071 scope.go:122] "RemoveContainer" containerID="77d8bf64718549233ad972f0369d7b4b6ac5b048e6e5fd33f60180b0ed2b245e"
Oct 30 21:00:17.081964 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:00:17.081945    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77d8bf64718549233ad972f0369d7b4b6ac5b048e6e5fd33f60180b0ed2b245e\": container with ID starting with 77d8bf64718549233ad972f0369d7b4b6ac5b048e6e5fd33f60180b0ed2b245e not found: ID does not exist" containerID="77d8bf64718549233ad972f0369d7b4b6ac5b048e6e5fd33f60180b0ed2b245e"
Oct 30 21:00:17.082048 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:17.081970    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77d8bf64718549233ad972f0369d7b4b6ac5b048e6e5fd33f60180b0ed2b245e"} err="failed to get container status \"77d8bf64718549233ad972f0369d7b4b6ac5b048e6e5fd33f60180b0ed2b245e\": rpc error: code = NotFound desc = could not find container \"77d8bf64718549233ad972f0369d7b4b6ac5b048e6e5fd33f60180b0ed2b245e\": container with ID starting with 77d8bf64718549233ad972f0369d7b4b6ac5b048e6e5fd33f60180b0ed2b245e not found: ID does not exist"
Oct 30 21:00:17.233064 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:17.233031    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-19176d-ff6f9d87d-5l9qg"]
Oct 30 21:00:17.236533 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:17.236511    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-19176d-ff6f9d87d-5l9qg"]
Oct 30 21:00:17.463292 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:17.463212    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-19176d-69f48c59bc-wgtb8"]
Oct 30 21:00:17.468767 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:17.468739    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-19176d-69f48c59bc-wgtb8"]
Oct 30 21:00:18.072715 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:18.072684    5071 generic.go:356] "Generic (PLEG): container finished" podID="72be9fd7-0964-4921-85d6-24c9f1609649" containerID="f39ad4cbf4c6975fd8e91a058c98549266fc5025f055b87e873e81af918e3768" exitCode=60
Oct 30 21:00:18.072923 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:18.072754    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8kndq" event={"ID":"72be9fd7-0964-4921-85d6-24c9f1609649","Type":"ContainerDied","Data":"f39ad4cbf4c6975fd8e91a058c98549266fc5025f055b87e873e81af918e3768"}
Oct 30 21:00:18.072923 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:18.072813    5071 scope.go:122] "RemoveContainer" containerID="5c0862ceffd0d5aeee49c866e524a5ea612b1fa13f9dc3de0536d3c937bc284e"
Oct 30 21:00:18.073210 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:18.073183    5071 scope.go:122] "RemoveContainer" containerID="f39ad4cbf4c6975fd8e91a058c98549266fc5025f055b87e873e81af918e3768"
Oct 30 21:00:18.073500 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:00:18.073476    5071 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889900-8kndq_odh-ai-gateway-infra(72be9fd7-0964-4921-85d6-24c9f1609649)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8kndq" podUID="72be9fd7-0964-4921-85d6-24c9f1609649"
Oct 30 21:00:18.211521 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:18.211480    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9104e5d7-09e8-4622-b783-69506c636052" path="/var/lib/kubelet/pods/9104e5d7-09e8-4622-b783-69506c636052/volumes"
Oct 30 21:00:18.211944 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:18.211926    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6a43db1-079d-4a4e-b048-641193df5ea4" path="/var/lib/kubelet/pods/b6a43db1-079d-4a4e-b048-641193df5ea4/volumes"
Oct 30 21:00:18.363906 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:18.363783    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8kndq"]
Oct 30 21:00:18.529368 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:18.529329    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-ee3fad-67dffbb55d-vjzhn" podUID="20a5960c-1432-48a8-8b82-730d6973647d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:00:18.529539 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:18.529329    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee3fad-69f6cc779f-plr9b" podUID="56a15d9e-6de3-47ab-8045-4a1c3618bc6a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:00:18.672041 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:18.671950    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:19.247947 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:19.247917    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8kndq"
Oct 30 21:00:19.283692 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:19.283666    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72be9fd7-0964-4921-85d6-24c9f1609649-kube-api-access-lljzj\" (UniqueName: \"kubernetes.io/projected/72be9fd7-0964-4921-85d6-24c9f1609649-kube-api-access-lljzj\") pod \"72be9fd7-0964-4921-85d6-24c9f1609649\" (UID: \"72be9fd7-0964-4921-85d6-24c9f1609649\") "
Oct 30 21:00:19.283888 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:19.283749    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72be9fd7-0964-4921-85d6-24c9f1609649-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72be9fd7-0964-4921-85d6-24c9f1609649-maas-api-service-ca\") pod \"72be9fd7-0964-4921-85d6-24c9f1609649\" (UID: \"72be9fd7-0964-4921-85d6-24c9f1609649\") "
Oct 30 21:00:19.284215 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:19.284102    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72be9fd7-0964-4921-85d6-24c9f1609649-maas-api-service-ca" pod "72be9fd7-0964-4921-85d6-24c9f1609649" (UID: "72be9fd7-0964-4921-85d6-24c9f1609649"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:19.286082 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:19.286061    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72be9fd7-0964-4921-85d6-24c9f1609649-kube-api-access-lljzj" pod "72be9fd7-0964-4921-85d6-24c9f1609649" (UID: "72be9fd7-0964-4921-85d6-24c9f1609649"). InnerVolumeSpecName "kube-api-access-lljzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:19.385991 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:19.385915    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lljzj\" (UniqueName: \"kubernetes.io/projected/72be9fd7-0964-4921-85d6-24c9f1609649-kube-api-access-lljzj\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:19.385991 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:19.385953    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72be9fd7-0964-4921-85d6-24c9f1609649-maas-api-service-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:20.090443 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:20.090413    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8kndq"
Oct 30 21:00:20.090655 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:20.090416    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8kndq" event={"ID":"72be9fd7-0964-4921-85d6-24c9f1609649","Type":"ContainerDied","Data":"b87166018932c35bbc6d44a67d8766d567918e70b8c5d09ec2406dd0f558913d"}
Oct 30 21:00:20.090655 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:20.090539    5071 scope.go:122] "RemoveContainer" containerID="f39ad4cbf4c6975fd8e91a058c98549266fc5025f055b87e873e81af918e3768"
Oct 30 21:00:20.255991 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:20.255953    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8kndq"]
Oct 30 21:00:20.305154 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:20.305119    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889900-8kndq"]
Oct 30 21:00:21.676243 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:21.676176    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"
Oct 30 21:00:21.709791 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:21.709726    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c80fd141-e109-4216-a3f1-39d326d72ab1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c80fd141-e109-4216-a3f1-39d326d72ab1-istio-podinfo\") pod \"c80fd141-e109-4216-a3f1-39d326d72ab1\" (UID: \"c80fd141-e109-4216-a3f1-39d326d72ab1\") "
Oct 30 21:00:21.709991 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:21.709857    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c80fd141-e109-4216-a3f1-39d326d72ab1-kube-api-access-tdlfz\" (UniqueName: \"kubernetes.io/projected/c80fd141-e109-4216-a3f1-39d326d72ab1-kube-api-access-tdlfz\") pod \"c80fd141-e109-4216-a3f1-39d326d72ab1\" (UID: \"c80fd141-e109-4216-a3f1-39d326d72ab1\") "
Oct 30 21:00:21.710056 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:21.709950    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c80fd141-e109-4216-a3f1-39d326d72ab1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c80fd141-e109-4216-a3f1-39d326d72ab1-istio-envoy\") pod \"c80fd141-e109-4216-a3f1-39d326d72ab1\" (UID: \"c80fd141-e109-4216-a3f1-39d326d72ab1\") "
Oct 30 21:00:21.710119 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:21.710074    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c80fd141-e109-4216-a3f1-39d326d72ab1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80fd141-e109-4216-a3f1-39d326d72ab1-credential-socket\") pod \"c80fd141-e109-4216-a3f1-39d326d72ab1\" (UID: \"c80fd141-e109-4216-a3f1-39d326d72ab1\") "
Oct 30 21:00:21.710187 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:21.710118    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c80fd141-e109-4216-a3f1-39d326d72ab1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c80fd141-e109-4216-a3f1-39d326d72ab1-istio-data\") pod \"c80fd141-e109-4216-a3f1-39d326d72ab1\" (UID: \"c80fd141-e109-4216-a3f1-39d326d72ab1\") "
Oct 30 21:00:21.710253 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:21.710230    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c80fd141-e109-4216-a3f1-39d326d72ab1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c80fd141-e109-4216-a3f1-39d326d72ab1-workload-certs\") pod \"c80fd141-e109-4216-a3f1-39d326d72ab1\" (UID: \"c80fd141-e109-4216-a3f1-39d326d72ab1\") "
Oct 30 21:00:21.710306 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:21.710290    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c80fd141-e109-4216-a3f1-39d326d72ab1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c80fd141-e109-4216-a3f1-39d326d72ab1-istiod-ca-cert\") pod \"c80fd141-e109-4216-a3f1-39d326d72ab1\" (UID: \"c80fd141-e109-4216-a3f1-39d326d72ab1\") "
Oct 30 21:00:21.710358 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:21.710330    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c80fd141-e109-4216-a3f1-39d326d72ab1-istio-token\" (UniqueName: \"kubernetes.io/projected/c80fd141-e109-4216-a3f1-39d326d72ab1-istio-token\") pod \"c80fd141-e109-4216-a3f1-39d326d72ab1\" (UID: \"c80fd141-e109-4216-a3f1-39d326d72ab1\") "
Oct 30 21:00:21.710358 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:21.710339    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80fd141-e109-4216-a3f1-39d326d72ab1-credential-socket" pod "c80fd141-e109-4216-a3f1-39d326d72ab1" (UID: "c80fd141-e109-4216-a3f1-39d326d72ab1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:21.710488 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:21.710375    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c80fd141-e109-4216-a3f1-39d326d72ab1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80fd141-e109-4216-a3f1-39d326d72ab1-workload-socket\") pod \"c80fd141-e109-4216-a3f1-39d326d72ab1\" (UID: \"c80fd141-e109-4216-a3f1-39d326d72ab1\") "
Oct 30 21:00:21.710753 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:21.710720    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80fd141-e109-4216-a3f1-39d326d72ab1-workload-certs" pod "c80fd141-e109-4216-a3f1-39d326d72ab1" (UID: "c80fd141-e109-4216-a3f1-39d326d72ab1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:21.710944 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:21.710853    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c80fd141-e109-4216-a3f1-39d326d72ab1-istiod-ca-cert" pod "c80fd141-e109-4216-a3f1-39d326d72ab1" (UID: "c80fd141-e109-4216-a3f1-39d326d72ab1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:21.710944 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:21.710879    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80fd141-e109-4216-a3f1-39d326d72ab1-istio-data" pod "c80fd141-e109-4216-a3f1-39d326d72ab1" (UID: "c80fd141-e109-4216-a3f1-39d326d72ab1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:21.711085 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:21.710971    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80fd141-e109-4216-a3f1-39d326d72ab1-workload-socket" pod "c80fd141-e109-4216-a3f1-39d326d72ab1" (UID: "c80fd141-e109-4216-a3f1-39d326d72ab1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:21.711085 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:21.711013    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c80fd141-e109-4216-a3f1-39d326d72ab1-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:21.711085 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:21.711030    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80fd141-e109-4216-a3f1-39d326d72ab1-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:21.712317 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:21.712288    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c80fd141-e109-4216-a3f1-39d326d72ab1-istio-podinfo" pod "c80fd141-e109-4216-a3f1-39d326d72ab1" (UID: "c80fd141-e109-4216-a3f1-39d326d72ab1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:00:21.712317 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:21.712307    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c80fd141-e109-4216-a3f1-39d326d72ab1-kube-api-access-tdlfz" pod "c80fd141-e109-4216-a3f1-39d326d72ab1" (UID: "c80fd141-e109-4216-a3f1-39d326d72ab1"). InnerVolumeSpecName "kube-api-access-tdlfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:21.712808 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:21.712776    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80fd141-e109-4216-a3f1-39d326d72ab1-istio-envoy" pod "c80fd141-e109-4216-a3f1-39d326d72ab1" (UID: "c80fd141-e109-4216-a3f1-39d326d72ab1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:21.712997 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:21.712942    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c80fd141-e109-4216-a3f1-39d326d72ab1-istio-token" pod "c80fd141-e109-4216-a3f1-39d326d72ab1" (UID: "c80fd141-e109-4216-a3f1-39d326d72ab1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:21.811898 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:21.811860    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c80fd141-e109-4216-a3f1-39d326d72ab1-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:21.811898 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:21.811890    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c80fd141-e109-4216-a3f1-39d326d72ab1-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:21.811898 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:21.811901    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c80fd141-e109-4216-a3f1-39d326d72ab1-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:21.812140 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:21.811910    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80fd141-e109-4216-a3f1-39d326d72ab1-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:21.812140 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:21.811920    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c80fd141-e109-4216-a3f1-39d326d72ab1-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:21.812140 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:21.811928    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdlfz\" (UniqueName: \"kubernetes.io/projected/c80fd141-e109-4216-a3f1-39d326d72ab1-kube-api-access-tdlfz\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:21.812140 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:21.811937    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c80fd141-e109-4216-a3f1-39d326d72ab1-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:22.105651 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:22.105615    5071 generic.go:356] "Generic (PLEG): container finished" podID="c80fd141-e109-4216-a3f1-39d326d72ab1" containerID="5d588107127690f85d2e4027d3b78fcf6e4b670f2f33679c83e80fbcdced4d8b" exitCode=0
Oct 30 21:00:22.105850 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:22.105668    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"
Oct 30 21:00:22.105850 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:22.105696    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9" event={"ID":"c80fd141-e109-4216-a3f1-39d326d72ab1","Type":"ContainerDied","Data":"5d588107127690f85d2e4027d3b78fcf6e4b670f2f33679c83e80fbcdced4d8b"}
Oct 30 21:00:22.105850 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:22.105748    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9" event={"ID":"c80fd141-e109-4216-a3f1-39d326d72ab1","Type":"ContainerDied","Data":"49b4c09795a48297c67f8d5e79d80d6e23683473a08db69aeeb2c69d0171f27e"}
Oct 30 21:00:22.105850 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:22.105771    5071 scope.go:122] "RemoveContainer" containerID="5d588107127690f85d2e4027d3b78fcf6e4b670f2f33679c83e80fbcdced4d8b"
Oct 30 21:00:22.116509 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:22.116490    5071 scope.go:122] "RemoveContainer" containerID="5d588107127690f85d2e4027d3b78fcf6e4b670f2f33679c83e80fbcdced4d8b"
Oct 30 21:00:22.116889 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:00:22.116864    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d588107127690f85d2e4027d3b78fcf6e4b670f2f33679c83e80fbcdced4d8b\": container with ID starting with 5d588107127690f85d2e4027d3b78fcf6e4b670f2f33679c83e80fbcdced4d8b not found: ID does not exist" containerID="5d588107127690f85d2e4027d3b78fcf6e4b670f2f33679c83e80fbcdced4d8b"
Oct 30 21:00:22.116992 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:22.116895    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d588107127690f85d2e4027d3b78fcf6e4b670f2f33679c83e80fbcdced4d8b"} err="failed to get container status \"5d588107127690f85d2e4027d3b78fcf6e4b670f2f33679c83e80fbcdced4d8b\": rpc error: code = NotFound desc = could not find container \"5d588107127690f85d2e4027d3b78fcf6e4b670f2f33679c83e80fbcdced4d8b\": container with ID starting with 5d588107127690f85d2e4027d3b78fcf6e4b670f2f33679c83e80fbcdced4d8b not found: ID does not exist"
Oct 30 21:00:22.200634 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:22.200582    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72be9fd7-0964-4921-85d6-24c9f1609649" path="/var/lib/kubelet/pods/72be9fd7-0964-4921-85d6-24c9f1609649/volumes"
Oct 30 21:00:22.273301 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:22.273254    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"]
Oct 30 21:00:22.322346 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:22.322307    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-386743-openshift-default-65c5f7bc77-g7pk9"]
Oct 30 21:00:23.016578 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:23.016546    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:24.067343 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:24.067310    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f"]
Oct 30 21:00:24.067778 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:24.067487    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f" podUID="0a7dd791-abaf-4b61-8fbf-25175613fb66" containerName="maas-api" containerID="cri-o://33a8247b2077fe7510522ed1ad751178f638b658b00e01eb27e52a9801fa4f85" gracePeriod=30
Oct 30 21:00:24.190373 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:24.190343    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:24.201118 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:24.201082    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c80fd141-e109-4216-a3f1-39d326d72ab1" path="/var/lib/kubelet/pods/c80fd141-e109-4216-a3f1-39d326d72ab1/volumes"
Oct 30 21:00:24.355967 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:24.355935    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f"
Oct 30 21:00:24.446456 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:24.446425    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a7dd791-abaf-4b61-8fbf-25175613fb66-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a7dd791-abaf-4b61-8fbf-25175613fb66-metrics-certs\") pod \"0a7dd791-abaf-4b61-8fbf-25175613fb66\" (UID: \"0a7dd791-abaf-4b61-8fbf-25175613fb66\") "
Oct 30 21:00:24.446634 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:24.446574    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a7dd791-abaf-4b61-8fbf-25175613fb66-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a7dd791-abaf-4b61-8fbf-25175613fb66-maas-api-tls\") pod \"0a7dd791-abaf-4b61-8fbf-25175613fb66\" (UID: \"0a7dd791-abaf-4b61-8fbf-25175613fb66\") "
Oct 30 21:00:24.446634 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:24.446618    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a7dd791-abaf-4b61-8fbf-25175613fb66-kube-api-access-vfcrd\" (UniqueName: \"kubernetes.io/projected/0a7dd791-abaf-4b61-8fbf-25175613fb66-kube-api-access-vfcrd\") pod \"0a7dd791-abaf-4b61-8fbf-25175613fb66\" (UID: \"0a7dd791-abaf-4b61-8fbf-25175613fb66\") "
Oct 30 21:00:24.446864 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:24.446773    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a7dd791-abaf-4b61-8fbf-25175613fb66-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a7dd791-abaf-4b61-8fbf-25175613fb66-openshift-service-ca\") pod \"0a7dd791-abaf-4b61-8fbf-25175613fb66\" (UID: \"0a7dd791-abaf-4b61-8fbf-25175613fb66\") "
Oct 30 21:00:24.447203 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:24.447170    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a7dd791-abaf-4b61-8fbf-25175613fb66-openshift-service-ca" pod "0a7dd791-abaf-4b61-8fbf-25175613fb66" (UID: "0a7dd791-abaf-4b61-8fbf-25175613fb66"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:24.448858 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:24.448796    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a7dd791-abaf-4b61-8fbf-25175613fb66-metrics-certs" pod "0a7dd791-abaf-4b61-8fbf-25175613fb66" (UID: "0a7dd791-abaf-4b61-8fbf-25175613fb66"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:24.448970 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:24.448888    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a7dd791-abaf-4b61-8fbf-25175613fb66-maas-api-tls" pod "0a7dd791-abaf-4b61-8fbf-25175613fb66" (UID: "0a7dd791-abaf-4b61-8fbf-25175613fb66"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:24.449066 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:24.449045    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a7dd791-abaf-4b61-8fbf-25175613fb66-kube-api-access-vfcrd" pod "0a7dd791-abaf-4b61-8fbf-25175613fb66" (UID: "0a7dd791-abaf-4b61-8fbf-25175613fb66"). InnerVolumeSpecName "kube-api-access-vfcrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:24.548658 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:24.548620    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfcrd\" (UniqueName: \"kubernetes.io/projected/0a7dd791-abaf-4b61-8fbf-25175613fb66-kube-api-access-vfcrd\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:24.548658 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:24.548650    5071 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a7dd791-abaf-4b61-8fbf-25175613fb66-openshift-service-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:24.548658 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:24.548660    5071 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a7dd791-abaf-4b61-8fbf-25175613fb66-metrics-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:24.548929 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:24.548670    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a7dd791-abaf-4b61-8fbf-25175613fb66-maas-api-tls\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:24.562012 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:24.561977    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:24.845536 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:24.845503    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:25.099755 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:25.099677    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee3fad-pjr8f"]
Oct 30 21:00:25.103531 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:25.103506    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ee3fad-pjr8f"]
Oct 30 21:00:25.143075 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:25.143024    5071 generic.go:356] "Generic (PLEG): container finished" podID="0a7dd791-abaf-4b61-8fbf-25175613fb66" containerID="33a8247b2077fe7510522ed1ad751178f638b658b00e01eb27e52a9801fa4f85" exitCode=0
Oct 30 21:00:25.143075 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:25.143076    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f"
Oct 30 21:00:25.143256 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:25.143109    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f" event={"ID":"0a7dd791-abaf-4b61-8fbf-25175613fb66","Type":"ContainerDied","Data":"33a8247b2077fe7510522ed1ad751178f638b658b00e01eb27e52a9801fa4f85"}
Oct 30 21:00:25.143256 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:25.143170    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f" event={"ID":"0a7dd791-abaf-4b61-8fbf-25175613fb66","Type":"ContainerDied","Data":"4909232f32052ac2af0e6e47c67f8375dbeb5d0fa6f35ee2399346f064c3626a"}
Oct 30 21:00:25.143256 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:25.143192    5071 scope.go:122] "RemoveContainer" containerID="33a8247b2077fe7510522ed1ad751178f638b658b00e01eb27e52a9801fa4f85"
Oct 30 21:00:25.154529 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:25.154509    5071 scope.go:122] "RemoveContainer" containerID="33a8247b2077fe7510522ed1ad751178f638b658b00e01eb27e52a9801fa4f85"
Oct 30 21:00:25.154872 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:00:25.154851    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33a8247b2077fe7510522ed1ad751178f638b658b00e01eb27e52a9801fa4f85\": container with ID starting with 33a8247b2077fe7510522ed1ad751178f638b658b00e01eb27e52a9801fa4f85 not found: ID does not exist" containerID="33a8247b2077fe7510522ed1ad751178f638b658b00e01eb27e52a9801fa4f85"
Oct 30 21:00:25.154950 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:25.154881    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a8247b2077fe7510522ed1ad751178f638b658b00e01eb27e52a9801fa4f85"} err="failed to get container status \"33a8247b2077fe7510522ed1ad751178f638b658b00e01eb27e52a9801fa4f85\": rpc error: code = NotFound desc = could not find container \"33a8247b2077fe7510522ed1ad751178f638b658b00e01eb27e52a9801fa4f85\": container with ID starting with 33a8247b2077fe7510522ed1ad751178f638b658b00e01eb27e52a9801fa4f85 not found: ID does not exist"
Oct 30 21:00:25.311318 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:25.311281    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f"]
Oct 30 21:00:25.317153 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:25.317127    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-19176d-7bd5d9979c-zkz6f"]
Oct 30 21:00:25.478953 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:25.478859    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:25.839395 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:25.839354    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:26.201477 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:26.201397    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a7dd791-abaf-4b61-8fbf-25175613fb66" path="/var/lib/kubelet/pods/0a7dd791-abaf-4b61-8fbf-25175613fb66/volumes"
Oct 30 21:00:26.201884 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:26.201808    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57013dd4-1ed6-4ebf-a118-c5ea00b8c087" path="/var/lib/kubelet/pods/57013dd4-1ed6-4ebf-a118-c5ea00b8c087/volumes"
Oct 30 21:00:26.229708 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:26.229676    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:26.395668 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:26.395628    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:26.466115 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:26.466038    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-19176d-fwph5"]
Oct 30 21:00:26.516861 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:26.516807    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-19176d-fwph5"]
Oct 30 21:00:26.906604 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:26.906552    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:28.200051 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:28.200021    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef4137c3-e3dc-4a70-b046-75a68d5ae460" path="/var/lib/kubelet/pods/ef4137c3-e3dc-4a70-b046-75a68d5ae460/volumes"
Oct 30 21:00:28.528934 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:28.528836    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-ee3fad-67dffbb55d-vjzhn" podUID="20a5960c-1432-48a8-8b82-730d6973647d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:00:28.528934 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:28.528870    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee3fad-69f6cc779f-plr9b" podUID="56a15d9e-6de3-47ab-8045-4a1c3618bc6a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:00:31.268173 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:31.268136    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:31.880864 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:31.880835    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-ee3fad-67dffbb55d-vjzhn"
Oct 30 21:00:31.981303 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:31.981275    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee3fad-69f6cc779f-plr9b"
Oct 30 21:00:32.024504 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:32.024471    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20a5960c-1432-48a8-8b82-730d6973647d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20a5960c-1432-48a8-8b82-730d6973647d-plugins-config-volume\") pod \"20a5960c-1432-48a8-8b82-730d6973647d\" (UID: \"20a5960c-1432-48a8-8b82-730d6973647d\") "
Oct 30 21:00:32.024715 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:32.024548    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20a5960c-1432-48a8-8b82-730d6973647d-kube-api-access-hbwhx\" (UniqueName: \"kubernetes.io/projected/20a5960c-1432-48a8-8b82-730d6973647d-kube-api-access-hbwhx\") pod \"20a5960c-1432-48a8-8b82-730d6973647d\" (UID: \"20a5960c-1432-48a8-8b82-730d6973647d\") "
Oct 30 21:00:32.025008 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:32.024979    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20a5960c-1432-48a8-8b82-730d6973647d-plugins-config-volume" pod "20a5960c-1432-48a8-8b82-730d6973647d" (UID: "20a5960c-1432-48a8-8b82-730d6973647d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:32.026916 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:32.026892    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a5960c-1432-48a8-8b82-730d6973647d-kube-api-access-hbwhx" pod "20a5960c-1432-48a8-8b82-730d6973647d" (UID: "20a5960c-1432-48a8-8b82-730d6973647d"). InnerVolumeSpecName "kube-api-access-hbwhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:32.050897 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:32.050868    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:32.126173 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:32.126081    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56a15d9e-6de3-47ab-8045-4a1c3618bc6a-kube-api-access-gwt9q\" (UniqueName: \"kubernetes.io/projected/56a15d9e-6de3-47ab-8045-4a1c3618bc6a-kube-api-access-gwt9q\") pod \"56a15d9e-6de3-47ab-8045-4a1c3618bc6a\" (UID: \"56a15d9e-6de3-47ab-8045-4a1c3618bc6a\") "
Oct 30 21:00:32.126173 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:32.126172    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56a15d9e-6de3-47ab-8045-4a1c3618bc6a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56a15d9e-6de3-47ab-8045-4a1c3618bc6a-plugins-config-volume\") pod \"56a15d9e-6de3-47ab-8045-4a1c3618bc6a\" (UID: \"56a15d9e-6de3-47ab-8045-4a1c3618bc6a\") "
Oct 30 21:00:32.126526 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:32.126509    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbwhx\" (UniqueName: \"kubernetes.io/projected/20a5960c-1432-48a8-8b82-730d6973647d-kube-api-access-hbwhx\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:32.126621 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:32.126529    5071 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20a5960c-1432-48a8-8b82-730d6973647d-plugins-config-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:32.126621 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:32.126578    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56a15d9e-6de3-47ab-8045-4a1c3618bc6a-plugins-config-volume" pod "56a15d9e-6de3-47ab-8045-4a1c3618bc6a" (UID: "56a15d9e-6de3-47ab-8045-4a1c3618bc6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:32.132059 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:32.132029    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a15d9e-6de3-47ab-8045-4a1c3618bc6a-kube-api-access-gwt9q" pod "56a15d9e-6de3-47ab-8045-4a1c3618bc6a" (UID: "56a15d9e-6de3-47ab-8045-4a1c3618bc6a"). InnerVolumeSpecName "kube-api-access-gwt9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:32.194600 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:32.194567    5071 generic.go:356] "Generic (PLEG): container finished" podID="20a5960c-1432-48a8-8b82-730d6973647d" containerID="2dd06967f87a7f0bb535f1b3b6f992a79f9cf637fde6bbe8d1089c6cb4bafe1e" exitCode=0
Oct 30 21:00:32.194807 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:32.194696    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-ee3fad-67dffbb55d-vjzhn"
Oct 30 21:00:32.207344 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:32.207297    5071 generic.go:356] "Generic (PLEG): container finished" podID="56a15d9e-6de3-47ab-8045-4a1c3618bc6a" containerID="14f626459dd48264a397c532e3c63cdf0dcbf06db5bcf38fd73177aa2b9eff50" exitCode=0
Oct 30 21:00:32.207531 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:32.207447    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee3fad-69f6cc779f-plr9b"
Oct 30 21:00:32.209498 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:32.209426    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-ee3fad-67dffbb55d-vjzhn" event={"ID":"20a5960c-1432-48a8-8b82-730d6973647d","Type":"ContainerDied","Data":"2dd06967f87a7f0bb535f1b3b6f992a79f9cf637fde6bbe8d1089c6cb4bafe1e"}
Oct 30 21:00:32.209644 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:32.209539    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-ee3fad-67dffbb55d-vjzhn" event={"ID":"20a5960c-1432-48a8-8b82-730d6973647d","Type":"ContainerDied","Data":"e9347ed3be2a07177c2ab66d7a24986817e894a33b3fc43edca7910c7d9a696d"}
Oct 30 21:00:32.209644 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:32.209578    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee3fad-69f6cc779f-plr9b" event={"ID":"56a15d9e-6de3-47ab-8045-4a1c3618bc6a","Type":"ContainerDied","Data":"14f626459dd48264a397c532e3c63cdf0dcbf06db5bcf38fd73177aa2b9eff50"}
Oct 30 21:00:32.209644 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:32.209610    5071 scope.go:122] "RemoveContainer" containerID="2dd06967f87a7f0bb535f1b3b6f992a79f9cf637fde6bbe8d1089c6cb4bafe1e"
Oct 30 21:00:32.209878 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:32.209780    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ee3fad-69f6cc779f-plr9b" event={"ID":"56a15d9e-6de3-47ab-8045-4a1c3618bc6a","Type":"ContainerDied","Data":"b3fbb288b6d513079fe05f4f80f81f833222e6420f351eaa5c774e519682c7be"}
Oct 30 21:00:32.221594 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:32.221574    5071 scope.go:122] "RemoveContainer" containerID="2dd06967f87a7f0bb535f1b3b6f992a79f9cf637fde6bbe8d1089c6cb4bafe1e"
Oct 30 21:00:32.221885 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:00:32.221865    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dd06967f87a7f0bb535f1b3b6f992a79f9cf637fde6bbe8d1089c6cb4bafe1e\": container with ID starting with 2dd06967f87a7f0bb535f1b3b6f992a79f9cf637fde6bbe8d1089c6cb4bafe1e not found: ID does not exist" containerID="2dd06967f87a7f0bb535f1b3b6f992a79f9cf637fde6bbe8d1089c6cb4bafe1e"
Oct 30 21:00:32.221975 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:32.221892    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd06967f87a7f0bb535f1b3b6f992a79f9cf637fde6bbe8d1089c6cb4bafe1e"} err="failed to get container status \"2dd06967f87a7f0bb535f1b3b6f992a79f9cf637fde6bbe8d1089c6cb4bafe1e\": rpc error: code = NotFound desc = could not find container \"2dd06967f87a7f0bb535f1b3b6f992a79f9cf637fde6bbe8d1089c6cb4bafe1e\": container with ID starting with 2dd06967f87a7f0bb535f1b3b6f992a79f9cf637fde6bbe8d1089c6cb4bafe1e not found: ID does not exist"
Oct 30 21:00:32.221975 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:32.221911    5071 scope.go:122] "RemoveContainer" containerID="14f626459dd48264a397c532e3c63cdf0dcbf06db5bcf38fd73177aa2b9eff50"
Oct 30 21:00:32.227164 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:32.227139    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwt9q\" (UniqueName: \"kubernetes.io/projected/56a15d9e-6de3-47ab-8045-4a1c3618bc6a-kube-api-access-gwt9q\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:32.227164 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:32.227162    5071 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56a15d9e-6de3-47ab-8045-4a1c3618bc6a-plugins-config-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:32.231318 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:32.231291    5071 scope.go:122] "RemoveContainer" containerID="14f626459dd48264a397c532e3c63cdf0dcbf06db5bcf38fd73177aa2b9eff50"
Oct 30 21:00:32.231627 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:00:32.231603    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14f626459dd48264a397c532e3c63cdf0dcbf06db5bcf38fd73177aa2b9eff50\": container with ID starting with 14f626459dd48264a397c532e3c63cdf0dcbf06db5bcf38fd73177aa2b9eff50 not found: ID does not exist" containerID="14f626459dd48264a397c532e3c63cdf0dcbf06db5bcf38fd73177aa2b9eff50"
Oct 30 21:00:32.231752 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:32.231638    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14f626459dd48264a397c532e3c63cdf0dcbf06db5bcf38fd73177aa2b9eff50"} err="failed to get container status \"14f626459dd48264a397c532e3c63cdf0dcbf06db5bcf38fd73177aa2b9eff50\": rpc error: code = NotFound desc = could not find container \"14f626459dd48264a397c532e3c63cdf0dcbf06db5bcf38fd73177aa2b9eff50\": container with ID starting with 14f626459dd48264a397c532e3c63cdf0dcbf06db5bcf38fd73177aa2b9eff50 not found: ID does not exist"
Oct 30 21:00:32.234394 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:32.234368    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:32.364904 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:32.364810    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-ee3fad-69f6cc779f-plr9b"]
Oct 30 21:00:32.379018 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:32.378934    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-ee3fad-69f6cc779f-plr9b"]
Oct 30 21:00:32.420427 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:32.420395    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:32.489431 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:32.488040    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-ee3fad-67dffbb55d-vjzhn"]
Oct 30 21:00:32.497814 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:32.496992    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-ee3fad-67dffbb55d-vjzhn"]
Oct 30 21:00:32.999693 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:32.999650    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:00:33.836374 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:33.836344    5071 scope.go:122] "RemoveContainer" containerID="2b3c96fb795b1d0bf730ea94433f9ac27af3f92ea1aa253ae2d5515a89283a98"
Oct 30 21:00:34.201203 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:34.201122    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20a5960c-1432-48a8-8b82-730d6973647d" path="/var/lib/kubelet/pods/20a5960c-1432-48a8-8b82-730d6973647d/volumes"
Oct 30 21:00:34.201529 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:34.201515    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56a15d9e-6de3-47ab-8045-4a1c3618bc6a" path="/var/lib/kubelet/pods/56a15d9e-6de3-47ab-8045-4a1c3618bc6a/volumes"
Oct 30 21:00:34.236924 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:34.236884    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:34.441017 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:34.440975    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:34.716312 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:34.716281    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"]
Oct 30 21:00:34.716527 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:34.716489    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4" podUID="e671c902-62e8-48fc-bf30-6a2505c31167" containerName="istio-proxy" containerID="cri-o://faada5b607d3b66e8af2f5e3a6b065d5b7ad94f82af1750e6a62f84ec913408a" gracePeriod=30
Oct 30 21:00:34.742041 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:34.742012    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:35.254571 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.254521    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:35.733176 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.733144    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:35.820264 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.820230    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-395022-2ntxl"]
Oct 30 21:00:35.820666 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.820654    5071 cpu_manager.go:414] "containerMap: removing container" podUID="72be9fd7-0964-4921-85d6-24c9f1609649" containerName="cleanup"
Oct 30 21:00:35.820707 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.820668    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72be9fd7-0964-4921-85d6-24c9f1609649" containerName="cleanup"
Oct 30 21:00:35.820707 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.820685    5071 cpu_manager.go:414] "containerMap: removing container" podUID="56a15d9e-6de3-47ab-8045-4a1c3618bc6a" containerName="payload-pre-processing"
Oct 30 21:00:35.820707 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.820691    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56a15d9e-6de3-47ab-8045-4a1c3618bc6a" containerName="payload-pre-processing"
Oct 30 21:00:35.820707 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.820705    5071 cpu_manager.go:414] "containerMap: removing container" podUID="72be9fd7-0964-4921-85d6-24c9f1609649" containerName="cleanup"
Oct 30 21:00:35.820861 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.820710    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72be9fd7-0964-4921-85d6-24c9f1609649" containerName="cleanup"
Oct 30 21:00:35.820861 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.820719    5071 cpu_manager.go:414] "containerMap: removing container" podUID="9104e5d7-09e8-4622-b783-69506c636052" containerName="payload-pre-processing"
Oct 30 21:00:35.820861 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.820724    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9104e5d7-09e8-4622-b783-69506c636052" containerName="payload-pre-processing"
Oct 30 21:00:35.820861 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.820732    5071 cpu_manager.go:414] "containerMap: removing container" podUID="c80fd141-e109-4216-a3f1-39d326d72ab1" containerName="istio-proxy"
Oct 30 21:00:35.820861 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.820736    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c80fd141-e109-4216-a3f1-39d326d72ab1" containerName="istio-proxy"
Oct 30 21:00:35.820861 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.820744    5071 cpu_manager.go:414] "containerMap: removing container" podUID="20a5960c-1432-48a8-8b82-730d6973647d" containerName="payload-processing"
Oct 30 21:00:35.820861 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.820748    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20a5960c-1432-48a8-8b82-730d6973647d" containerName="payload-processing"
Oct 30 21:00:35.820861 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.820762    5071 cpu_manager.go:414] "containerMap: removing container" podUID="72be9fd7-0964-4921-85d6-24c9f1609649" containerName="cleanup"
Oct 30 21:00:35.820861 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.820766    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72be9fd7-0964-4921-85d6-24c9f1609649" containerName="cleanup"
Oct 30 21:00:35.820861 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.820773    5071 cpu_manager.go:414] "containerMap: removing container" podUID="b6a43db1-079d-4a4e-b048-641193df5ea4" containerName="payload-processing"
Oct 30 21:00:35.820861 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.820777    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6a43db1-079d-4a4e-b048-641193df5ea4" containerName="payload-processing"
Oct 30 21:00:35.820861 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.820786    5071 cpu_manager.go:414] "containerMap: removing container" podUID="6dc29059-37fb-4f25-8809-148d6115a8a5" containerName="revoke-keys"
Oct 30 21:00:35.820861 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.820790    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dc29059-37fb-4f25-8809-148d6115a8a5" containerName="revoke-keys"
Oct 30 21:00:35.820861 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.820797    5071 cpu_manager.go:414] "containerMap: removing container" podUID="0a7dd791-abaf-4b61-8fbf-25175613fb66" containerName="maas-api"
Oct 30 21:00:35.820861 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.820801    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a7dd791-abaf-4b61-8fbf-25175613fb66" containerName="maas-api"
Oct 30 21:00:35.821346 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.820881    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="72be9fd7-0964-4921-85d6-24c9f1609649" containerName="cleanup"
Oct 30 21:00:35.821346 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.820890    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="56a15d9e-6de3-47ab-8045-4a1c3618bc6a" containerName="payload-pre-processing"
Oct 30 21:00:35.821346 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.820898    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="9104e5d7-09e8-4622-b783-69506c636052" containerName="payload-pre-processing"
Oct 30 21:00:35.821346 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.820906    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="c80fd141-e109-4216-a3f1-39d326d72ab1" containerName="istio-proxy"
Oct 30 21:00:35.821346 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.820913    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="20a5960c-1432-48a8-8b82-730d6973647d" containerName="payload-processing"
Oct 30 21:00:35.821346 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.820918    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="70eb560e-dc33-4be9-a107-a2bed8967828" containerName="cleanup"
Oct 30 21:00:35.821346 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.820926    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="72be9fd7-0964-4921-85d6-24c9f1609649" containerName="cleanup"
Oct 30 21:00:35.821346 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.820931    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6a43db1-079d-4a4e-b048-641193df5ea4" containerName="payload-processing"
Oct 30 21:00:35.821346 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.820937    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dc29059-37fb-4f25-8809-148d6115a8a5" containerName="revoke-keys"
Oct 30 21:00:35.821346 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.820943    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a7dd791-abaf-4b61-8fbf-25175613fb66" containerName="maas-api"
Oct 30 21:00:35.825982 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.825962    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-395022-2ntxl"
Oct 30 21:00:35.881900 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.881874    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-395022-2ntxl"]
Oct 30 21:00:35.928444 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.928390    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:00:35.928444 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.928390    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:00:35.928444 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.928447    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:00:35.928707 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.928518    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 21:00:35.968260 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.968218    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38e4e675-5d3e-413c-86a0-3b6c43fcad74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-395022-2ntxl\" (UID: \"38e4e675-5d3e-413c-86a0-3b6c43fcad74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-395022-2ntxl"
Oct 30 21:00:35.968459 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.968268    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd96m\" (UniqueName: \"kubernetes.io/projected/38e4e675-5d3e-413c-86a0-3b6c43fcad74-kube-api-access-vd96m\") pod \"maas-api-revoke-keys-e2e-worker-w4-395022-2ntxl\" (UID: \"38e4e675-5d3e-413c-86a0-3b6c43fcad74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-395022-2ntxl"
Oct 30 21:00:35.968459 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.968410    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38e4e675-5d3e-413c-86a0-3b6c43fcad74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-395022-2ntxl\" (UID: \"38e4e675-5d3e-413c-86a0-3b6c43fcad74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-395022-2ntxl"
Oct 30 21:00:35.985670 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:35.985569    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:36.069323 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:36.069292    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38e4e675-5d3e-413c-86a0-3b6c43fcad74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-395022-2ntxl\" (UID: \"38e4e675-5d3e-413c-86a0-3b6c43fcad74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-395022-2ntxl"
Oct 30 21:00:36.069323 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:36.069326    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd96m\" (UniqueName: \"kubernetes.io/projected/38e4e675-5d3e-413c-86a0-3b6c43fcad74-kube-api-access-vd96m\") pod \"maas-api-revoke-keys-e2e-worker-w4-395022-2ntxl\" (UID: \"38e4e675-5d3e-413c-86a0-3b6c43fcad74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-395022-2ntxl"
Oct 30 21:00:36.069568 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:36.069375    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38e4e675-5d3e-413c-86a0-3b6c43fcad74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-395022-2ntxl\" (UID: \"38e4e675-5d3e-413c-86a0-3b6c43fcad74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-395022-2ntxl"
Oct 30 21:00:36.070177 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:36.070154    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38e4e675-5d3e-413c-86a0-3b6c43fcad74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-395022-2ntxl\" (UID: \"38e4e675-5d3e-413c-86a0-3b6c43fcad74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-395022-2ntxl"
Oct 30 21:00:36.072118 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:36.072090    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38e4e675-5d3e-413c-86a0-3b6c43fcad74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-395022-2ntxl\" (UID: \"38e4e675-5d3e-413c-86a0-3b6c43fcad74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-395022-2ntxl"
Oct 30 21:00:36.128132 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:36.128099    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd96m\" (UniqueName: \"kubernetes.io/projected/38e4e675-5d3e-413c-86a0-3b6c43fcad74-kube-api-access-vd96m\") pod \"maas-api-revoke-keys-e2e-worker-w4-395022-2ntxl\" (UID: \"38e4e675-5d3e-413c-86a0-3b6c43fcad74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-395022-2ntxl"
Oct 30 21:00:36.141987 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:36.141956    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-395022-2ntxl"
Oct 30 21:00:36.156186 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:36.156156    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:36.473780 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:36.473738    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:36.514967 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:36.514933    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-395022-2ntxl"]
Oct 30 21:00:36.516335 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:00:36.516309    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38e4e675_5d3e_413c_86a0_3b6c43fcad74.slice/crio-b4c7a0153d35d260b031493021448cf58eb4b6de095ed29e963fbffeac483ffd WatchSource:0}: Error finding container b4c7a0153d35d260b031493021448cf58eb4b6de095ed29e963fbffeac483ffd: Status 404 returned error can't find the container with id b4c7a0153d35d260b031493021448cf58eb4b6de095ed29e963fbffeac483ffd
Oct 30 21:00:37.239051 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:37.239018    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-395022-2ntxl" event={"ID":"38e4e675-5d3e-413c-86a0-3b6c43fcad74","Type":"ContainerStarted","Data":"598e1dea3d58f6d8f42acb85156d12cb6729954811abb8084dd1efff6b424ddc"}
Oct 30 21:00:37.239221 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:37.239098    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-395022-2ntxl" event={"ID":"38e4e675-5d3e-413c-86a0-3b6c43fcad74","Type":"ContainerStarted","Data":"b4c7a0153d35d260b031493021448cf58eb4b6de095ed29e963fbffeac483ffd"}
Oct 30 21:00:37.356880 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:37.356808    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-395022-2ntxl" podStartSLOduration=2.3567949710000002 podStartE2EDuration="2.356794971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:37.354345065 +0000 UTC m=+2049.271366633" watchObservedRunningTime="2026-10-30 21:00:37.356794971 +0000 UTC m=+2049.273816507"
Oct 30 21:00:38.246110 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:38.246073    5071 generic.go:356] "Generic (PLEG): container finished" podID="38e4e675-5d3e-413c-86a0-3b6c43fcad74" containerID="598e1dea3d58f6d8f42acb85156d12cb6729954811abb8084dd1efff6b424ddc" exitCode=0
Oct 30 21:00:38.246523 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:38.246155    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-395022-2ntxl" event={"ID":"38e4e675-5d3e-413c-86a0-3b6c43fcad74","Type":"ContainerDied","Data":"598e1dea3d58f6d8f42acb85156d12cb6729954811abb8084dd1efff6b424ddc"}
Oct 30 21:00:38.675317 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:38.675278    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:39.404254 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:39.404228    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-395022-2ntxl"
Oct 30 21:00:39.506025 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:39.505933    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38e4e675-5d3e-413c-86a0-3b6c43fcad74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38e4e675-5d3e-413c-86a0-3b6c43fcad74-maas-api-xks-ca\") pod \"38e4e675-5d3e-413c-86a0-3b6c43fcad74\" (UID: \"38e4e675-5d3e-413c-86a0-3b6c43fcad74\") "
Oct 30 21:00:39.506025 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:39.505997    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38e4e675-5d3e-413c-86a0-3b6c43fcad74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38e4e675-5d3e-413c-86a0-3b6c43fcad74-maas-api-openshift-ca\") pod \"38e4e675-5d3e-413c-86a0-3b6c43fcad74\" (UID: \"38e4e675-5d3e-413c-86a0-3b6c43fcad74\") "
Oct 30 21:00:39.506271 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:39.506066    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38e4e675-5d3e-413c-86a0-3b6c43fcad74-kube-api-access-vd96m\" (UniqueName: \"kubernetes.io/projected/38e4e675-5d3e-413c-86a0-3b6c43fcad74-kube-api-access-vd96m\") pod \"38e4e675-5d3e-413c-86a0-3b6c43fcad74\" (UID: \"38e4e675-5d3e-413c-86a0-3b6c43fcad74\") "
Oct 30 21:00:39.506379 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:39.506352    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38e4e675-5d3e-413c-86a0-3b6c43fcad74-maas-api-openshift-ca" pod "38e4e675-5d3e-413c-86a0-3b6c43fcad74" (UID: "38e4e675-5d3e-413c-86a0-3b6c43fcad74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:39.508351 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:39.508327    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38e4e675-5d3e-413c-86a0-3b6c43fcad74-maas-api-xks-ca" pod "38e4e675-5d3e-413c-86a0-3b6c43fcad74" (UID: "38e4e675-5d3e-413c-86a0-3b6c43fcad74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:39.508458 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:39.508389    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38e4e675-5d3e-413c-86a0-3b6c43fcad74-kube-api-access-vd96m" pod "38e4e675-5d3e-413c-86a0-3b6c43fcad74" (UID: "38e4e675-5d3e-413c-86a0-3b6c43fcad74"). InnerVolumeSpecName "kube-api-access-vd96m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:39.607834 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:39.607796    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38e4e675-5d3e-413c-86a0-3b6c43fcad74-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:39.608017 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:39.607846    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38e4e675-5d3e-413c-86a0-3b6c43fcad74-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:39.608017 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:39.607861    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd96m\" (UniqueName: \"kubernetes.io/projected/38e4e675-5d3e-413c-86a0-3b6c43fcad74-kube-api-access-vd96m\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:39.972873 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:39.972846    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 21:00:40.113319 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.113233    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e671c902-62e8-48fc-bf30-6a2505c31167-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e671c902-62e8-48fc-bf30-6a2505c31167-workload-certs\") pod \"e671c902-62e8-48fc-bf30-6a2505c31167\" (UID: \"e671c902-62e8-48fc-bf30-6a2505c31167\") "
Oct 30 21:00:40.113319 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.113302    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e671c902-62e8-48fc-bf30-6a2505c31167-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e671c902-62e8-48fc-bf30-6a2505c31167-istiod-ca-cert\") pod \"e671c902-62e8-48fc-bf30-6a2505c31167\" (UID: \"e671c902-62e8-48fc-bf30-6a2505c31167\") "
Oct 30 21:00:40.113558 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.113338    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e671c902-62e8-48fc-bf30-6a2505c31167-kube-api-access-5z6tw\" (UniqueName: \"kubernetes.io/projected/e671c902-62e8-48fc-bf30-6a2505c31167-kube-api-access-5z6tw\") pod \"e671c902-62e8-48fc-bf30-6a2505c31167\" (UID: \"e671c902-62e8-48fc-bf30-6a2505c31167\") "
Oct 30 21:00:40.113558 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.113375    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e671c902-62e8-48fc-bf30-6a2505c31167-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e671c902-62e8-48fc-bf30-6a2505c31167-istio-data\") pod \"e671c902-62e8-48fc-bf30-6a2505c31167\" (UID: \"e671c902-62e8-48fc-bf30-6a2505c31167\") "
Oct 30 21:00:40.113558 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.113402    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e671c902-62e8-48fc-bf30-6a2505c31167-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e671c902-62e8-48fc-bf30-6a2505c31167-istio-envoy\") pod \"e671c902-62e8-48fc-bf30-6a2505c31167\" (UID: \"e671c902-62e8-48fc-bf30-6a2505c31167\") "
Oct 30 21:00:40.113558 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.113429    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e671c902-62e8-48fc-bf30-6a2505c31167-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e671c902-62e8-48fc-bf30-6a2505c31167-workload-socket\") pod \"e671c902-62e8-48fc-bf30-6a2505c31167\" (UID: \"e671c902-62e8-48fc-bf30-6a2505c31167\") "
Oct 30 21:00:40.113558 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.113498    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e671c902-62e8-48fc-bf30-6a2505c31167-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e671c902-62e8-48fc-bf30-6a2505c31167-istio-podinfo\") pod \"e671c902-62e8-48fc-bf30-6a2505c31167\" (UID: \"e671c902-62e8-48fc-bf30-6a2505c31167\") "
Oct 30 21:00:40.113847 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.113577    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e671c902-62e8-48fc-bf30-6a2505c31167-workload-certs" pod "e671c902-62e8-48fc-bf30-6a2505c31167" (UID: "e671c902-62e8-48fc-bf30-6a2505c31167"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:40.113847 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.113690    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e671c902-62e8-48fc-bf30-6a2505c31167-istio-data" pod "e671c902-62e8-48fc-bf30-6a2505c31167" (UID: "e671c902-62e8-48fc-bf30-6a2505c31167"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:40.113847 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.113777    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e671c902-62e8-48fc-bf30-6a2505c31167-workload-socket" pod "e671c902-62e8-48fc-bf30-6a2505c31167" (UID: "e671c902-62e8-48fc-bf30-6a2505c31167"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:40.113847 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.113782    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e671c902-62e8-48fc-bf30-6a2505c31167-istiod-ca-cert" pod "e671c902-62e8-48fc-bf30-6a2505c31167" (UID: "e671c902-62e8-48fc-bf30-6a2505c31167"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:40.114061 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.113897    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e671c902-62e8-48fc-bf30-6a2505c31167-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e671c902-62e8-48fc-bf30-6a2505c31167-credential-socket\") pod \"e671c902-62e8-48fc-bf30-6a2505c31167\" (UID: \"e671c902-62e8-48fc-bf30-6a2505c31167\") "
Oct 30 21:00:40.114061 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.113947    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e671c902-62e8-48fc-bf30-6a2505c31167-istio-token\" (UniqueName: \"kubernetes.io/projected/e671c902-62e8-48fc-bf30-6a2505c31167-istio-token\") pod \"e671c902-62e8-48fc-bf30-6a2505c31167\" (UID: \"e671c902-62e8-48fc-bf30-6a2505c31167\") "
Oct 30 21:00:40.114152 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.114113    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e671c902-62e8-48fc-bf30-6a2505c31167-credential-socket" pod "e671c902-62e8-48fc-bf30-6a2505c31167" (UID: "e671c902-62e8-48fc-bf30-6a2505c31167"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:40.114369 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.114351    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e671c902-62e8-48fc-bf30-6a2505c31167-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:40.114444 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.114373    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e671c902-62e8-48fc-bf30-6a2505c31167-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:40.114444 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.114387    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e671c902-62e8-48fc-bf30-6a2505c31167-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:40.114444 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.114401    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e671c902-62e8-48fc-bf30-6a2505c31167-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:40.114444 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.114414    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e671c902-62e8-48fc-bf30-6a2505c31167-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:40.116119 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.116098    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e671c902-62e8-48fc-bf30-6a2505c31167-istio-token" pod "e671c902-62e8-48fc-bf30-6a2505c31167" (UID: "e671c902-62e8-48fc-bf30-6a2505c31167"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:40.116388 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.116359    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e671c902-62e8-48fc-bf30-6a2505c31167-istio-envoy" pod "e671c902-62e8-48fc-bf30-6a2505c31167" (UID: "e671c902-62e8-48fc-bf30-6a2505c31167"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:40.116478 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.116418    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e671c902-62e8-48fc-bf30-6a2505c31167-kube-api-access-5z6tw" pod "e671c902-62e8-48fc-bf30-6a2505c31167" (UID: "e671c902-62e8-48fc-bf30-6a2505c31167"). InnerVolumeSpecName "kube-api-access-5z6tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:40.116478 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.116423    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e671c902-62e8-48fc-bf30-6a2505c31167-istio-podinfo" pod "e671c902-62e8-48fc-bf30-6a2505c31167" (UID: "e671c902-62e8-48fc-bf30-6a2505c31167"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:00:40.192943 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.192902    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"]
Oct 30 21:00:40.201412 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.201387    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"]
Oct 30 21:00:40.215114 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.215088    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z6tw\" (UniqueName: \"kubernetes.io/projected/e671c902-62e8-48fc-bf30-6a2505c31167-kube-api-access-5z6tw\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:40.215114 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.215115    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e671c902-62e8-48fc-bf30-6a2505c31167-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:40.215294 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.215125    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e671c902-62e8-48fc-bf30-6a2505c31167-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:40.215294 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.215133    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e671c902-62e8-48fc-bf30-6a2505c31167-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:40.258722 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.258696    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-395022-2ntxl"
Oct 30 21:00:40.258901 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.258733    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-395022-2ntxl" event={"ID":"38e4e675-5d3e-413c-86a0-3b6c43fcad74","Type":"ContainerDied","Data":"b4c7a0153d35d260b031493021448cf58eb4b6de095ed29e963fbffeac483ffd"}
Oct 30 21:00:40.258901 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.258790    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4c7a0153d35d260b031493021448cf58eb4b6de095ed29e963fbffeac483ffd"
Oct 30 21:00:40.261073 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.261030    5071 generic.go:356] "Generic (PLEG): container finished" podID="e671c902-62e8-48fc-bf30-6a2505c31167" containerID="faada5b607d3b66e8af2f5e3a6b065d5b7ad94f82af1750e6a62f84ec913408a" exitCode=0
Oct 30 21:00:40.261184 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.261164    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4" event={"ID":"e671c902-62e8-48fc-bf30-6a2505c31167","Type":"ContainerDied","Data":"faada5b607d3b66e8af2f5e3a6b065d5b7ad94f82af1750e6a62f84ec913408a"}
Oct 30 21:00:40.261260 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.261209    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4" event={"ID":"e671c902-62e8-48fc-bf30-6a2505c31167","Type":"ContainerDied","Data":"122aaf70397c99a78bf308aa9a6b8cc5c1150d31ba57ec1e445cfedadb808984"}
Oct 30 21:00:40.261260 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.261239    5071 scope.go:122] "RemoveContainer" containerID="faada5b607d3b66e8af2f5e3a6b065d5b7ad94f82af1750e6a62f84ec913408a"
Oct 30 21:00:40.261381 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.261289    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"
Oct 30 21:00:40.272083 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.272053    5071 scope.go:122] "RemoveContainer" containerID="7154b1841877e313e01bf6c52cd7ed097f0b93e299dc64d12725273d337fdff7"
Oct 30 21:00:40.282084 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.282062    5071 scope.go:122] "RemoveContainer" containerID="faada5b607d3b66e8af2f5e3a6b065d5b7ad94f82af1750e6a62f84ec913408a"
Oct 30 21:00:40.282384 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:00:40.282353    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faada5b607d3b66e8af2f5e3a6b065d5b7ad94f82af1750e6a62f84ec913408a\": container with ID starting with faada5b607d3b66e8af2f5e3a6b065d5b7ad94f82af1750e6a62f84ec913408a not found: ID does not exist" containerID="faada5b607d3b66e8af2f5e3a6b065d5b7ad94f82af1750e6a62f84ec913408a"
Oct 30 21:00:40.282483 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.282394    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faada5b607d3b66e8af2f5e3a6b065d5b7ad94f82af1750e6a62f84ec913408a"} err="failed to get container status \"faada5b607d3b66e8af2f5e3a6b065d5b7ad94f82af1750e6a62f84ec913408a\": rpc error: code = NotFound desc = could not find container \"faada5b607d3b66e8af2f5e3a6b065d5b7ad94f82af1750e6a62f84ec913408a\": container with ID starting with faada5b607d3b66e8af2f5e3a6b065d5b7ad94f82af1750e6a62f84ec913408a not found: ID does not exist"
Oct 30 21:00:40.282483 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.282414    5071 scope.go:122] "RemoveContainer" containerID="7154b1841877e313e01bf6c52cd7ed097f0b93e299dc64d12725273d337fdff7"
Oct 30 21:00:40.282743 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:00:40.282695    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7154b1841877e313e01bf6c52cd7ed097f0b93e299dc64d12725273d337fdff7\": container with ID starting with 7154b1841877e313e01bf6c52cd7ed097f0b93e299dc64d12725273d337fdff7 not found: ID does not exist" containerID="7154b1841877e313e01bf6c52cd7ed097f0b93e299dc64d12725273d337fdff7"
Oct 30 21:00:40.282795 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.282753    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7154b1841877e313e01bf6c52cd7ed097f0b93e299dc64d12725273d337fdff7"} err="failed to get container status \"7154b1841877e313e01bf6c52cd7ed097f0b93e299dc64d12725273d337fdff7\": rpc error: code = NotFound desc = could not find container \"7154b1841877e313e01bf6c52cd7ed097f0b93e299dc64d12725273d337fdff7\": container with ID starting with 7154b1841877e313e01bf6c52cd7ed097f0b93e299dc64d12725273d337fdff7 not found: ID does not exist"
Oct 30 21:00:40.381584 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.381481    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:40.474660 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.474616    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"]
Oct 30 21:00:40.482988 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:40.482947    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-19176d-openshift-default-6bf869759c-s2kq4"]
Oct 30 21:00:41.149036 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:41.148995    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:41.473928 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:41.473808    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:42.200635 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:42.200603    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e671c902-62e8-48fc-bf30-6a2505c31167" path="/var/lib/kubelet/pods/e671c902-62e8-48fc-bf30-6a2505c31167/volumes"
Oct 30 21:00:44.975463 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:44.975419    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:46.487056 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:46.487012    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:46.727447 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:00:46.727411    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:00:47.495166 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:47.495114    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-395022-5d767b7967-6nf69"]
Oct 30 21:00:47.778056 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:47.777976    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:47.942229 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:47.942197    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:50.231367 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:50.231332    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:50.390145 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:50.390116    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:51.383346 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:51.383284    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:00:52.496924 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:52.496883    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-395022-5d767b7967-6nf69" podUID="6262b3d5-64e5-4e9c-a4b9-a08f25298572" containerName="payload-pre-processing" containerID="cri-o://c1e97d01d1ac68576dadef9d6de7a2a1f2cd9431c29b0cd327e8c18463ceb380" gracePeriod=25
Oct 30 21:00:53.568680 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:53.568643    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-395022-5d767b7967-6nf69" podUID="6262b3d5-64e5-4e9c-a4b9-a08f25298572" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:00:55.215041 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.214980    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main" containerID="cri-o://b11501b47f868d4440aef56c6fa201025614da58454209e738244333f0bb8d42" gracePeriod=45
Oct 30 21:00:55.224227 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.224174    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main" containerID="cri-o://4974a19737d331919de23e6f3ff07bd90ab7a0b1ff4b6bb64986d5edd1ce354f" gracePeriod=45
Oct 30 21:00:55.350851 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.350797    5071 generic.go:356] "Generic (PLEG): container finished" podID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerID="b11501b47f868d4440aef56c6fa201025614da58454209e738244333f0bb8d42" exitCode=0
Oct 30 21:00:55.351015 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.350876    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" event={"ID":"1b9bf342-553e-4f8b-aa1c-71fd95b70b1f","Type":"ContainerDied","Data":"b11501b47f868d4440aef56c6fa201025614da58454209e738244333f0bb8d42"}
Oct 30 21:00:55.353052 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.352970    5071 generic.go:356] "Generic (PLEG): container finished" podID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerID="4974a19737d331919de23e6f3ff07bd90ab7a0b1ff4b6bb64986d5edd1ce354f" exitCode=0
Oct 30 21:00:55.353052 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.353021    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" event={"ID":"7919394d-614c-4f2d-b04a-f44f90d56e96","Type":"ContainerDied","Data":"4974a19737d331919de23e6f3ff07bd90ab7a0b1ff4b6bb64986d5edd1ce354f"}
Oct 30 21:00:55.503940 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.503912    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 21:00:55.529232 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.529210    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 21:00:55.569574 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.569542    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7919394d-614c-4f2d-b04a-f44f90d56e96-tls-certs\" (UniqueName: \"kubernetes.io/secret/7919394d-614c-4f2d-b04a-f44f90d56e96-tls-certs\") pod \"7919394d-614c-4f2d-b04a-f44f90d56e96\" (UID: \"7919394d-614c-4f2d-b04a-f44f90d56e96\") "
Oct 30 21:00:55.569733 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.569590    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-model-cache\") pod \"1b9bf342-553e-4f8b-aa1c-71fd95b70b1f\" (UID: \"1b9bf342-553e-4f8b-aa1c-71fd95b70b1f\") "
Oct 30 21:00:55.569733 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.569615    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7919394d-614c-4f2d-b04a-f44f90d56e96-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7919394d-614c-4f2d-b04a-f44f90d56e96-dshm\") pod \"7919394d-614c-4f2d-b04a-f44f90d56e96\" (UID: \"7919394d-614c-4f2d-b04a-f44f90d56e96\") "
Oct 30 21:00:55.569733 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.569669    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7919394d-614c-4f2d-b04a-f44f90d56e96-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7919394d-614c-4f2d-b04a-f44f90d56e96-model-cache\") pod \"7919394d-614c-4f2d-b04a-f44f90d56e96\" (UID: \"7919394d-614c-4f2d-b04a-f44f90d56e96\") "
Oct 30 21:00:55.569733 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.569714    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-home\" (UniqueName: \"kubernetes.io/empty-dir/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-home\") pod \"1b9bf342-553e-4f8b-aa1c-71fd95b70b1f\" (UID: \"1b9bf342-553e-4f8b-aa1c-71fd95b70b1f\") "
Oct 30 21:00:55.569986 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.569783    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-kube-api-access-qp645\" (UniqueName: \"kubernetes.io/projected/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-kube-api-access-qp645\") pod \"1b9bf342-553e-4f8b-aa1c-71fd95b70b1f\" (UID: \"1b9bf342-553e-4f8b-aa1c-71fd95b70b1f\") "
Oct 30 21:00:55.569986 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.569876    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7919394d-614c-4f2d-b04a-f44f90d56e96-kube-api-access-wx4ch\" (UniqueName: \"kubernetes.io/projected/7919394d-614c-4f2d-b04a-f44f90d56e96-kube-api-access-wx4ch\") pod \"7919394d-614c-4f2d-b04a-f44f90d56e96\" (UID: \"7919394d-614c-4f2d-b04a-f44f90d56e96\") "
Oct 30 21:00:55.569986 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.569903    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-model-cache" pod "1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" (UID: "1b9bf342-553e-4f8b-aa1c-71fd95b70b1f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:55.569986 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.569932    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7919394d-614c-4f2d-b04a-f44f90d56e96-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7919394d-614c-4f2d-b04a-f44f90d56e96-tmp-dir\") pod \"7919394d-614c-4f2d-b04a-f44f90d56e96\" (UID: \"7919394d-614c-4f2d-b04a-f44f90d56e96\") "
Oct 30 21:00:55.570181 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.569992    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-tmp-dir\") pod \"1b9bf342-553e-4f8b-aa1c-71fd95b70b1f\" (UID: \"1b9bf342-553e-4f8b-aa1c-71fd95b70b1f\") "
Oct 30 21:00:55.570181 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.570034    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-tls-certs\" (UniqueName: \"kubernetes.io/secret/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-tls-certs\") pod \"1b9bf342-553e-4f8b-aa1c-71fd95b70b1f\" (UID: \"1b9bf342-553e-4f8b-aa1c-71fd95b70b1f\") "
Oct 30 21:00:55.570181 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.570063    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-dshm\") pod \"1b9bf342-553e-4f8b-aa1c-71fd95b70b1f\" (UID: \"1b9bf342-553e-4f8b-aa1c-71fd95b70b1f\") "
Oct 30 21:00:55.570181 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.570099    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7919394d-614c-4f2d-b04a-f44f90d56e96-home\" (UniqueName: \"kubernetes.io/empty-dir/7919394d-614c-4f2d-b04a-f44f90d56e96-home\") pod \"7919394d-614c-4f2d-b04a-f44f90d56e96\" (UID: \"7919394d-614c-4f2d-b04a-f44f90d56e96\") "
Oct 30 21:00:55.570516 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.570492    5071 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-model-cache\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:55.571600 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.570895    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7919394d-614c-4f2d-b04a-f44f90d56e96-home" pod "7919394d-614c-4f2d-b04a-f44f90d56e96" (UID: "7919394d-614c-4f2d-b04a-f44f90d56e96"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:55.571600 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.570958    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7919394d-614c-4f2d-b04a-f44f90d56e96-tmp-dir" pod "7919394d-614c-4f2d-b04a-f44f90d56e96" (UID: "7919394d-614c-4f2d-b04a-f44f90d56e96"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:55.571600 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.571146    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7919394d-614c-4f2d-b04a-f44f90d56e96-model-cache" pod "7919394d-614c-4f2d-b04a-f44f90d56e96" (UID: "7919394d-614c-4f2d-b04a-f44f90d56e96"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:55.571600 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.571232    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-home" pod "1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" (UID: "1b9bf342-553e-4f8b-aa1c-71fd95b70b1f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:55.571600 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.571477    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-tmp-dir" pod "1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" (UID: "1b9bf342-553e-4f8b-aa1c-71fd95b70b1f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:55.572935 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.572897    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7919394d-614c-4f2d-b04a-f44f90d56e96-tls-certs" pod "7919394d-614c-4f2d-b04a-f44f90d56e96" (UID: "7919394d-614c-4f2d-b04a-f44f90d56e96"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:55.573104 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.572983    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7919394d-614c-4f2d-b04a-f44f90d56e96-dshm" pod "7919394d-614c-4f2d-b04a-f44f90d56e96" (UID: "7919394d-614c-4f2d-b04a-f44f90d56e96"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:55.573206 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.573182    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7919394d-614c-4f2d-b04a-f44f90d56e96-kube-api-access-wx4ch" pod "7919394d-614c-4f2d-b04a-f44f90d56e96" (UID: "7919394d-614c-4f2d-b04a-f44f90d56e96"). InnerVolumeSpecName "kube-api-access-wx4ch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:55.573354 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.573336    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-dshm" pod "1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" (UID: "1b9bf342-553e-4f8b-aa1c-71fd95b70b1f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:55.573426 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.573354    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-tls-certs" pod "1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" (UID: "1b9bf342-553e-4f8b-aa1c-71fd95b70b1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:55.573899 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.573871    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-kube-api-access-qp645" pod "1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" (UID: "1b9bf342-553e-4f8b-aa1c-71fd95b70b1f"). InnerVolumeSpecName "kube-api-access-qp645". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:55.671666 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.671634    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx4ch\" (UniqueName: \"kubernetes.io/projected/7919394d-614c-4f2d-b04a-f44f90d56e96-kube-api-access-wx4ch\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:55.671666 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.671666    5071 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7919394d-614c-4f2d-b04a-f44f90d56e96-tmp-dir\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:55.671884 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.671676    5071 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-tmp-dir\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:55.671884 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.671684    5071 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-tls-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:55.671884 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.671692    5071 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-dshm\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:55.671884 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.671699    5071 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7919394d-614c-4f2d-b04a-f44f90d56e96-home\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:55.671884 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.671706    5071 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7919394d-614c-4f2d-b04a-f44f90d56e96-tls-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:55.671884 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.671714    5071 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7919394d-614c-4f2d-b04a-f44f90d56e96-dshm\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:55.671884 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.671721    5071 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7919394d-614c-4f2d-b04a-f44f90d56e96-model-cache\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:55.671884 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.671730    5071 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-home\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:55.671884 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.671753    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp645\" (UniqueName: \"kubernetes.io/projected/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f-kube-api-access-qp645\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:55.674651 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.674616    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-280ee9d9-rww2g"]
Oct 30 21:00:55.675074 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.675060    5071 cpu_manager.go:414] "containerMap: removing container" podUID="38e4e675-5d3e-413c-86a0-3b6c43fcad74" containerName="revoke-keys"
Oct 30 21:00:55.675130 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.675075    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38e4e675-5d3e-413c-86a0-3b6c43fcad74" containerName="revoke-keys"
Oct 30 21:00:55.675130 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.675092    5071 cpu_manager.go:414] "containerMap: removing container" podUID="e671c902-62e8-48fc-bf30-6a2505c31167" containerName="istio-proxy"
Oct 30 21:00:55.675130 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.675097    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e671c902-62e8-48fc-bf30-6a2505c31167" containerName="istio-proxy"
Oct 30 21:00:55.675130 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.675107    5071 cpu_manager.go:414] "containerMap: removing container" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main"
Oct 30 21:00:55.675130 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.675112    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main"
Oct 30 21:00:55.675130 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.675125    5071 cpu_manager.go:414] "containerMap: removing container" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main"
Oct 30 21:00:55.675130 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.675130    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main"
Oct 30 21:00:55.675347 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.675202    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="e671c902-62e8-48fc-bf30-6a2505c31167" containerName="istio-proxy"
Oct 30 21:00:55.675347 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.675211    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="72be9fd7-0964-4921-85d6-24c9f1609649" containerName="cleanup"
Oct 30 21:00:55.675347 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.675217    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="38e4e675-5d3e-413c-86a0-3b6c43fcad74" containerName="revoke-keys"
Oct 30 21:00:55.675347 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.675223    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="e671c902-62e8-48fc-bf30-6a2505c31167" containerName="istio-proxy"
Oct 30 21:00:55.675347 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.675229    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" containerName="main"
Oct 30 21:00:55.675347 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.675235    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" containerName="main"
Oct 30 21:00:55.679402 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.679382    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-280ee9d9-rww2g"
Oct 30 21:00:55.733726 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.733659    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 21:00:55.735398 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.735379    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:00:55.736801 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.736558    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:00:55.736801 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.736656    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:00:55.738483 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.738444    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-280ee9d9-rww2g"]
Oct 30 21:00:55.772539 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.772504    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsclj\" (UniqueName: \"kubernetes.io/projected/cdef71a8-3a7d-4d75-bb49-6bb0e4da9437-kube-api-access-fsclj\") pod \"maas-api-revoke-sub-280ee9d9-rww2g\" (UID: \"cdef71a8-3a7d-4d75-bb49-6bb0e4da9437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-280ee9d9-rww2g"
Oct 30 21:00:55.772749 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.772560    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdef71a8-3a7d-4d75-bb49-6bb0e4da9437-maas-api-xks-ca\") pod \"maas-api-revoke-sub-280ee9d9-rww2g\" (UID: \"cdef71a8-3a7d-4d75-bb49-6bb0e4da9437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-280ee9d9-rww2g"
Oct 30 21:00:55.772749 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.772698    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdef71a8-3a7d-4d75-bb49-6bb0e4da9437-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-280ee9d9-rww2g\" (UID: \"cdef71a8-3a7d-4d75-bb49-6bb0e4da9437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-280ee9d9-rww2g"
Oct 30 21:00:55.874256 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.874218    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsclj\" (UniqueName: \"kubernetes.io/projected/cdef71a8-3a7d-4d75-bb49-6bb0e4da9437-kube-api-access-fsclj\") pod \"maas-api-revoke-sub-280ee9d9-rww2g\" (UID: \"cdef71a8-3a7d-4d75-bb49-6bb0e4da9437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-280ee9d9-rww2g"
Oct 30 21:00:55.874458 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.874269    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdef71a8-3a7d-4d75-bb49-6bb0e4da9437-maas-api-xks-ca\") pod \"maas-api-revoke-sub-280ee9d9-rww2g\" (UID: \"cdef71a8-3a7d-4d75-bb49-6bb0e4da9437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-280ee9d9-rww2g"
Oct 30 21:00:55.874458 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.874321    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdef71a8-3a7d-4d75-bb49-6bb0e4da9437-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-280ee9d9-rww2g\" (UID: \"cdef71a8-3a7d-4d75-bb49-6bb0e4da9437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-280ee9d9-rww2g"
Oct 30 21:00:55.875018 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.874991    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdef71a8-3a7d-4d75-bb49-6bb0e4da9437-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-280ee9d9-rww2g\" (UID: \"cdef71a8-3a7d-4d75-bb49-6bb0e4da9437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-280ee9d9-rww2g"
Oct 30 21:00:55.876846 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.876811    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdef71a8-3a7d-4d75-bb49-6bb0e4da9437-maas-api-xks-ca\") pod \"maas-api-revoke-sub-280ee9d9-rww2g\" (UID: \"cdef71a8-3a7d-4d75-bb49-6bb0e4da9437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-280ee9d9-rww2g"
Oct 30 21:00:55.932174 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.932139    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsclj\" (UniqueName: \"kubernetes.io/projected/cdef71a8-3a7d-4d75-bb49-6bb0e4da9437-kube-api-access-fsclj\") pod \"maas-api-revoke-sub-280ee9d9-rww2g\" (UID: \"cdef71a8-3a7d-4d75-bb49-6bb0e4da9437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-280ee9d9-rww2g"
Oct 30 21:00:55.992759 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:55.992690    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-280ee9d9-rww2g"
Oct 30 21:00:56.359933 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:56.359899    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"
Oct 30 21:00:56.360437 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:56.359936    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr" event={"ID":"7919394d-614c-4f2d-b04a-f44f90d56e96","Type":"ContainerDied","Data":"e454e2d83d28b90d243709bf7133fb4437f34457e63a4d5bf75e4d65470dbc6e"}
Oct 30 21:00:56.360437 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:56.359989    5071 scope.go:122] "RemoveContainer" containerID="4974a19737d331919de23e6f3ff07bd90ab7a0b1ff4b6bb64986d5edd1ce354f"
Oct 30 21:00:56.361924 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:56.361800    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw" event={"ID":"1b9bf342-553e-4f8b-aa1c-71fd95b70b1f","Type":"ContainerDied","Data":"b7984a9e21acbbc32eab71382f7bb3f0c154416471d9541766c34a250bb8aa8d"}
Oct 30 21:00:56.362006 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:56.361857    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"
Oct 30 21:00:56.367373 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:56.367353    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-280ee9d9-rww2g"]
Oct 30 21:00:56.369290 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:00:56.369268    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdef71a8_3a7d_4d75_bb49_6bb0e4da9437.slice/crio-887ef26fff3808de5d3944fa013778964b4717094f359c7a9c1844f3a66b222d WatchSource:0}: Error finding container 887ef26fff3808de5d3944fa013778964b4717094f359c7a9c1844f3a66b222d: Status 404 returned error can't find the container with id 887ef26fff3808de5d3944fa013778964b4717094f359c7a9c1844f3a66b222d
Oct 30 21:00:56.372073 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:56.372049    5071 scope.go:122] "RemoveContainer" containerID="b11501b47f868d4440aef56c6fa201025614da58454209e738244333f0bb8d42"
Oct 30 21:00:56.525099 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:56.525069    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"]
Oct 30 21:00:56.531449 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:56.530768    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-395022/distinct-w4-395022-kserve-77f46f9b8d-sn9tw"]
Oct 30 21:00:56.640549 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:56.640524    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"]
Oct 30 21:00:56.645711 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:56.645685    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-395022/premium-w4-395022-kserve-5d7d5658f7-kgjvr"]
Oct 30 21:00:57.371021 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:57.370983    5071 generic.go:356] "Generic (PLEG): container finished" podID="cdef71a8-3a7d-4d75-bb49-6bb0e4da9437" containerID="961e19dd805717aa5fcc892ab6781e7b1c30499572da7915a6910274cd947f34" exitCode=0
Oct 30 21:00:57.371418 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:57.371078    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-280ee9d9-rww2g" event={"ID":"cdef71a8-3a7d-4d75-bb49-6bb0e4da9437","Type":"ContainerDied","Data":"961e19dd805717aa5fcc892ab6781e7b1c30499572da7915a6910274cd947f34"}
Oct 30 21:00:57.371418 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:57.371125    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-280ee9d9-rww2g" event={"ID":"cdef71a8-3a7d-4d75-bb49-6bb0e4da9437","Type":"ContainerStarted","Data":"887ef26fff3808de5d3944fa013778964b4717094f359c7a9c1844f3a66b222d"}
Oct 30 21:00:58.201779 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:58.201749    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b9bf342-553e-4f8b-aa1c-71fd95b70b1f" path="/var/lib/kubelet/pods/1b9bf342-553e-4f8b-aa1c-71fd95b70b1f/volumes"
Oct 30 21:00:58.202185 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:58.202171    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7919394d-614c-4f2d-b04a-f44f90d56e96" path="/var/lib/kubelet/pods/7919394d-614c-4f2d-b04a-f44f90d56e96/volumes"
Oct 30 21:00:58.521846 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:58.521803    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-280ee9d9-rww2g"
Oct 30 21:00:58.608596 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:58.608554    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdef71a8-3a7d-4d75-bb49-6bb0e4da9437-kube-api-access-fsclj\" (UniqueName: \"kubernetes.io/projected/cdef71a8-3a7d-4d75-bb49-6bb0e4da9437-kube-api-access-fsclj\") pod \"cdef71a8-3a7d-4d75-bb49-6bb0e4da9437\" (UID: \"cdef71a8-3a7d-4d75-bb49-6bb0e4da9437\") "
Oct 30 21:00:58.608800 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:58.608651    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdef71a8-3a7d-4d75-bb49-6bb0e4da9437-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdef71a8-3a7d-4d75-bb49-6bb0e4da9437-maas-api-openshift-ca\") pod \"cdef71a8-3a7d-4d75-bb49-6bb0e4da9437\" (UID: \"cdef71a8-3a7d-4d75-bb49-6bb0e4da9437\") "
Oct 30 21:00:58.608800 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:58.608754    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdef71a8-3a7d-4d75-bb49-6bb0e4da9437-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdef71a8-3a7d-4d75-bb49-6bb0e4da9437-maas-api-xks-ca\") pod \"cdef71a8-3a7d-4d75-bb49-6bb0e4da9437\" (UID: \"cdef71a8-3a7d-4d75-bb49-6bb0e4da9437\") "
Oct 30 21:00:58.609071 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:58.609047    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdef71a8-3a7d-4d75-bb49-6bb0e4da9437-maas-api-openshift-ca" pod "cdef71a8-3a7d-4d75-bb49-6bb0e4da9437" (UID: "cdef71a8-3a7d-4d75-bb49-6bb0e4da9437"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:58.609161 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:58.609141    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdef71a8-3a7d-4d75-bb49-6bb0e4da9437-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:58.610997 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:58.610963    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdef71a8-3a7d-4d75-bb49-6bb0e4da9437-kube-api-access-fsclj" pod "cdef71a8-3a7d-4d75-bb49-6bb0e4da9437" (UID: "cdef71a8-3a7d-4d75-bb49-6bb0e4da9437"). InnerVolumeSpecName "kube-api-access-fsclj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:58.611126 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:58.611018    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdef71a8-3a7d-4d75-bb49-6bb0e4da9437-maas-api-xks-ca" pod "cdef71a8-3a7d-4d75-bb49-6bb0e4da9437" (UID: "cdef71a8-3a7d-4d75-bb49-6bb0e4da9437"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:58.710029 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:58.709998    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsclj\" (UniqueName: \"kubernetes.io/projected/cdef71a8-3a7d-4d75-bb49-6bb0e4da9437-kube-api-access-fsclj\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:58.710029 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:58.710027    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdef71a8-3a7d-4d75-bb49-6bb0e4da9437-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:59.384483 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:59.384455    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-280ee9d9-rww2g"
Oct 30 21:00:59.384689 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:59.384482    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-280ee9d9-rww2g" event={"ID":"cdef71a8-3a7d-4d75-bb49-6bb0e4da9437","Type":"ContainerDied","Data":"887ef26fff3808de5d3944fa013778964b4717094f359c7a9c1844f3a66b222d"}
Oct 30 21:00:59.384689 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:00:59.384528    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="887ef26fff3808de5d3944fa013778964b4717094f359c7a9c1844f3a66b222d"
Oct 30 21:01:03.569345 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:03.569304    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-395022-5d767b7967-6nf69" podUID="6262b3d5-64e5-4e9c-a4b9-a08f25298572" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:01:11.149286 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:11.149247    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:12.186070 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:12.186038    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:01:12.776281 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:12.776254    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-395022-5d767b7967-6nf69"
Oct 30 21:01:12.778215 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:12.778191    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6262b3d5-64e5-4e9c-a4b9-a08f25298572-kube-api-access-mxdlr\" (UniqueName: \"kubernetes.io/projected/6262b3d5-64e5-4e9c-a4b9-a08f25298572-kube-api-access-mxdlr\") pod \"6262b3d5-64e5-4e9c-a4b9-a08f25298572\" (UID: \"6262b3d5-64e5-4e9c-a4b9-a08f25298572\") "
Oct 30 21:01:12.778358 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:12.778231    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6262b3d5-64e5-4e9c-a4b9-a08f25298572-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6262b3d5-64e5-4e9c-a4b9-a08f25298572-plugins-config-volume\") pod \"6262b3d5-64e5-4e9c-a4b9-a08f25298572\" (UID: \"6262b3d5-64e5-4e9c-a4b9-a08f25298572\") "
Oct 30 21:01:12.778738 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:12.778708    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6262b3d5-64e5-4e9c-a4b9-a08f25298572-plugins-config-volume" pod "6262b3d5-64e5-4e9c-a4b9-a08f25298572" (UID: "6262b3d5-64e5-4e9c-a4b9-a08f25298572"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:12.780405 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:12.780383    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6262b3d5-64e5-4e9c-a4b9-a08f25298572-kube-api-access-mxdlr" pod "6262b3d5-64e5-4e9c-a4b9-a08f25298572" (UID: "6262b3d5-64e5-4e9c-a4b9-a08f25298572"). InnerVolumeSpecName "kube-api-access-mxdlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:12.879461 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:12.879427    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxdlr\" (UniqueName: \"kubernetes.io/projected/6262b3d5-64e5-4e9c-a4b9-a08f25298572-kube-api-access-mxdlr\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:12.879461 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:12.879456    5071 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6262b3d5-64e5-4e9c-a4b9-a08f25298572-plugins-config-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:13.213606 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:13.213576    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-395022-2ntxl"]
Oct 30 21:01:13.217263 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:13.217239    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-395022-2ntxl"]
Oct 30 21:01:13.390033 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:13.389993    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:13.460320 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:13.460290    5071 generic.go:356] "Generic (PLEG): container finished" podID="6262b3d5-64e5-4e9c-a4b9-a08f25298572" containerID="c1e97d01d1ac68576dadef9d6de7a2a1f2cd9431c29b0cd327e8c18463ceb380" exitCode=0
Oct 30 21:01:13.460500 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:13.460339    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-395022-5d767b7967-6nf69"
Oct 30 21:01:13.460500 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:13.460369    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-395022-5d767b7967-6nf69" event={"ID":"6262b3d5-64e5-4e9c-a4b9-a08f25298572","Type":"ContainerDied","Data":"c1e97d01d1ac68576dadef9d6de7a2a1f2cd9431c29b0cd327e8c18463ceb380"}
Oct 30 21:01:13.460500 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:13.460430    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-395022-5d767b7967-6nf69" event={"ID":"6262b3d5-64e5-4e9c-a4b9-a08f25298572","Type":"ContainerDied","Data":"2c7626ca7b1109b360c85002a901997ba79f43bf0f133e8aa475a15b4c49c399"}
Oct 30 21:01:13.460500 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:13.460459    5071 scope.go:122] "RemoveContainer" containerID="c1e97d01d1ac68576dadef9d6de7a2a1f2cd9431c29b0cd327e8c18463ceb380"
Oct 30 21:01:13.471474 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:13.471453    5071 scope.go:122] "RemoveContainer" containerID="c1e97d01d1ac68576dadef9d6de7a2a1f2cd9431c29b0cd327e8c18463ceb380"
Oct 30 21:01:13.471779 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:01:13.471758    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1e97d01d1ac68576dadef9d6de7a2a1f2cd9431c29b0cd327e8c18463ceb380\": container with ID starting with c1e97d01d1ac68576dadef9d6de7a2a1f2cd9431c29b0cd327e8c18463ceb380 not found: ID does not exist" containerID="c1e97d01d1ac68576dadef9d6de7a2a1f2cd9431c29b0cd327e8c18463ceb380"
Oct 30 21:01:13.471878 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:13.471789    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1e97d01d1ac68576dadef9d6de7a2a1f2cd9431c29b0cd327e8c18463ceb380"} err="failed to get container status \"c1e97d01d1ac68576dadef9d6de7a2a1f2cd9431c29b0cd327e8c18463ceb380\": rpc error: code = NotFound desc = could not find container \"c1e97d01d1ac68576dadef9d6de7a2a1f2cd9431c29b0cd327e8c18463ceb380\": container with ID starting with c1e97d01d1ac68576dadef9d6de7a2a1f2cd9431c29b0cd327e8c18463ceb380 not found: ID does not exist"
Oct 30 21:01:13.628685 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:13.628648    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-395022-5d767b7967-6nf69"]
Oct 30 21:01:13.677173 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:13.677140    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-395022-5d767b7967-6nf69"]
Oct 30 21:01:13.687797 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:13.687753    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:14.201336 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:14.201294    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38e4e675-5d3e-413c-86a0-3b6c43fcad74" path="/var/lib/kubelet/pods/38e4e675-5d3e-413c-86a0-3b6c43fcad74/volumes"
Oct 30 21:01:14.201754 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:14.201732    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6262b3d5-64e5-4e9c-a4b9-a08f25298572" path="/var/lib/kubelet/pods/6262b3d5-64e5-4e9c-a4b9-a08f25298572/volumes"
Oct 30 21:01:16.727298 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:01:16.727254    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:01:17.661213 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:17.661173    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:18.506379 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:18.506342    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:18.719730 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:18.719694    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:20.221791 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:20.221752    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:21.502264 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:21.502226    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:21.664774 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:21.664736    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:21.878607 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:21.878569    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:26.558694 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:26.558649    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:27.193292 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:27.193254    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6565584cfb-w8rp4"]
Oct 30 21:01:27.193580 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:27.193526    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6565584cfb-w8rp4" podUID="b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607" containerName="manager" containerID="cri-o://e88f539c1d7621283f4029f206bdb6d628705eba5a59a3b5d965f29a1368ba4a" gracePeriod=30
Oct 30 21:01:27.463674 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:27.463648    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6565584cfb-w8rp4"
Oct 30 21:01:27.526053 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:27.526019    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607-metrics-certs\") pod \"b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607\" (UID: \"b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607\") "
Oct 30 21:01:27.526236 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:27.526072    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607-webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607-webhook-cert\") pod \"b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607\" (UID: \"b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607\") "
Oct 30 21:01:27.526236 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:27.526110    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607-kube-api-access-5tsvl\" (UniqueName: \"kubernetes.io/projected/b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607-kube-api-access-5tsvl\") pod \"b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607\" (UID: \"b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607\") "
Oct 30 21:01:27.528297 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:27.528262    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607-webhook-cert" pod "b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607" (UID: "b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:27.528416 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:27.528302    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607-metrics-certs" pod "b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607" (UID: "b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:27.528416 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:27.528320    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607-kube-api-access-5tsvl" pod "b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607" (UID: "b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607"). InnerVolumeSpecName "kube-api-access-5tsvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:27.541391 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:27.541363    5071 generic.go:356] "Generic (PLEG): container finished" podID="b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607" containerID="e88f539c1d7621283f4029f206bdb6d628705eba5a59a3b5d965f29a1368ba4a" exitCode=0
Oct 30 21:01:27.541504 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:27.541416    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6565584cfb-w8rp4"
Oct 30 21:01:27.541504 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:27.541444    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6565584cfb-w8rp4" event={"ID":"b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607","Type":"ContainerDied","Data":"e88f539c1d7621283f4029f206bdb6d628705eba5a59a3b5d965f29a1368ba4a"}
Oct 30 21:01:27.541629 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:27.541507    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6565584cfb-w8rp4" event={"ID":"b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607","Type":"ContainerDied","Data":"47dc5b1498c12482d873f500e1201846a0bcaffc1dba81308f8a29369ee86440"}
Oct 30 21:01:27.541629 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:27.541538    5071 scope.go:122] "RemoveContainer" containerID="e88f539c1d7621283f4029f206bdb6d628705eba5a59a3b5d965f29a1368ba4a"
Oct 30 21:01:27.551129 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:27.551113    5071 scope.go:122] "RemoveContainer" containerID="d1edfe6aec2489767e8aa2e1c677191ae5cc7c382ab8e800953737d207cf3e61"
Oct 30 21:01:27.559344 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:27.559318    5071 scope.go:122] "RemoveContainer" containerID="e88f539c1d7621283f4029f206bdb6d628705eba5a59a3b5d965f29a1368ba4a"
Oct 30 21:01:27.559678 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:01:27.559596    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e88f539c1d7621283f4029f206bdb6d628705eba5a59a3b5d965f29a1368ba4a\": container with ID starting with e88f539c1d7621283f4029f206bdb6d628705eba5a59a3b5d965f29a1368ba4a not found: ID does not exist" containerID="e88f539c1d7621283f4029f206bdb6d628705eba5a59a3b5d965f29a1368ba4a"
Oct 30 21:01:27.559678 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:27.559648    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e88f539c1d7621283f4029f206bdb6d628705eba5a59a3b5d965f29a1368ba4a"} err="failed to get container status \"e88f539c1d7621283f4029f206bdb6d628705eba5a59a3b5d965f29a1368ba4a\": rpc error: code = NotFound desc = could not find container \"e88f539c1d7621283f4029f206bdb6d628705eba5a59a3b5d965f29a1368ba4a\": container with ID starting with e88f539c1d7621283f4029f206bdb6d628705eba5a59a3b5d965f29a1368ba4a not found: ID does not exist"
Oct 30 21:01:27.559678 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:27.559667    5071 scope.go:122] "RemoveContainer" containerID="d1edfe6aec2489767e8aa2e1c677191ae5cc7c382ab8e800953737d207cf3e61"
Oct 30 21:01:27.559919 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:01:27.559897    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1edfe6aec2489767e8aa2e1c677191ae5cc7c382ab8e800953737d207cf3e61\": container with ID starting with d1edfe6aec2489767e8aa2e1c677191ae5cc7c382ab8e800953737d207cf3e61 not found: ID does not exist" containerID="d1edfe6aec2489767e8aa2e1c677191ae5cc7c382ab8e800953737d207cf3e61"
Oct 30 21:01:27.559990 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:27.559928    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1edfe6aec2489767e8aa2e1c677191ae5cc7c382ab8e800953737d207cf3e61"} err="failed to get container status \"d1edfe6aec2489767e8aa2e1c677191ae5cc7c382ab8e800953737d207cf3e61\": rpc error: code = NotFound desc = could not find container \"d1edfe6aec2489767e8aa2e1c677191ae5cc7c382ab8e800953737d207cf3e61\": container with ID starting with d1edfe6aec2489767e8aa2e1c677191ae5cc7c382ab8e800953737d207cf3e61 not found: ID does not exist"
Oct 30 21:01:27.584221 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:27.584194    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:01:27.627244 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:27.627204    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tsvl\" (UniqueName: \"kubernetes.io/projected/b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607-kube-api-access-5tsvl\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:27.627244 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:27.627234    5071 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607-metrics-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:27.627244 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:27.627250    5071 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607-webhook-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:27.694751 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:27.694695    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6565584cfb-w8rp4"]
Oct 30 21:01:27.698358 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:27.698294    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6565584cfb-w8rp4"]
Oct 30 21:01:28.202487 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:28.202448    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607" path="/var/lib/kubelet/pods/b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607/volumes"
Oct 30 21:01:28.618442 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:28.618408    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6565584cfb-4hgzm"]
Oct 30 21:01:28.619112 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:28.619090    5071 cpu_manager.go:414] "containerMap: removing container" podUID="6262b3d5-64e5-4e9c-a4b9-a08f25298572" containerName="payload-pre-processing"
Oct 30 21:01:28.619270 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:28.619114    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6262b3d5-64e5-4e9c-a4b9-a08f25298572" containerName="payload-pre-processing"
Oct 30 21:01:28.619270 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:28.619139    5071 cpu_manager.go:414] "containerMap: removing container" podUID="e671c902-62e8-48fc-bf30-6a2505c31167" containerName="istio-proxy"
Oct 30 21:01:28.619270 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:28.619147    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e671c902-62e8-48fc-bf30-6a2505c31167" containerName="istio-proxy"
Oct 30 21:01:28.619270 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:28.619162    5071 cpu_manager.go:414] "containerMap: removing container" podUID="b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607" containerName="manager"
Oct 30 21:01:28.619270 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:28.619169    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607" containerName="manager"
Oct 30 21:01:28.619270 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:28.619188    5071 cpu_manager.go:414] "containerMap: removing container" podUID="cdef71a8-3a7d-4d75-bb49-6bb0e4da9437" containerName="revoke-keys"
Oct 30 21:01:28.619270 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:28.619194    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdef71a8-3a7d-4d75-bb49-6bb0e4da9437" containerName="revoke-keys"
Oct 30 21:01:28.619641 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:28.619303    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="6262b3d5-64e5-4e9c-a4b9-a08f25298572" containerName="payload-pre-processing"
Oct 30 21:01:28.619641 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:28.619319    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607" containerName="manager"
Oct 30 21:01:28.619641 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:28.619332    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607" containerName="manager"
Oct 30 21:01:28.619641 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:28.619340    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdef71a8-3a7d-4d75-bb49-6bb0e4da9437" containerName="revoke-keys"
Oct 30 21:01:28.624117 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:28.624095    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6565584cfb-4hgzm"
Oct 30 21:01:28.726931 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:28.726861    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sk69l\""
Oct 30 21:01:28.726931 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:28.726881    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 21:01:28.726931 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:28.726884    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 21:01:28.727291 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:28.726951    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 21:01:28.730775 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:28.730748    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6565584cfb-4hgzm"]
Oct 30 21:01:28.740467 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:28.740434    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d799739f-553c-4245-8ee7-4742ba6b84bf-metrics-certs\") pod \"maas-controller-6565584cfb-4hgzm\" (UID: \"d799739f-553c-4245-8ee7-4742ba6b84bf\") " pod="opendatahub/maas-controller-6565584cfb-4hgzm"
Oct 30 21:01:28.740648 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:28.740496    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d799739f-553c-4245-8ee7-4742ba6b84bf-webhook-cert\") pod \"maas-controller-6565584cfb-4hgzm\" (UID: \"d799739f-553c-4245-8ee7-4742ba6b84bf\") " pod="opendatahub/maas-controller-6565584cfb-4hgzm"
Oct 30 21:01:28.740648 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:28.740526    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mvgm\" (UniqueName: \"kubernetes.io/projected/d799739f-553c-4245-8ee7-4742ba6b84bf-kube-api-access-2mvgm\") pod \"maas-controller-6565584cfb-4hgzm\" (UID: \"d799739f-553c-4245-8ee7-4742ba6b84bf\") " pod="opendatahub/maas-controller-6565584cfb-4hgzm"
Oct 30 21:01:28.757738 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:28.757704    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8qvb_91fe9e06-0c30-4000-bddb-2ba89cd048cc/console-operator/2.log"
Oct 30 21:01:28.771202 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:28.771146    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q4f4h_80cae79e-b60f-4efe-bb58-e77753a457e5/ovn-acl-logging/0.log"
Oct 30 21:01:28.775321 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:28.775299    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8qvb_91fe9e06-0c30-4000-bddb-2ba89cd048cc/console-operator/2.log"
Oct 30 21:01:28.789021 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:28.788995    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q4f4h_80cae79e-b60f-4efe-bb58-e77753a457e5/ovn-acl-logging/0.log"
Oct 30 21:01:28.848247 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:28.848201    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d799739f-553c-4245-8ee7-4742ba6b84bf-metrics-certs\") pod \"maas-controller-6565584cfb-4hgzm\" (UID: \"d799739f-553c-4245-8ee7-4742ba6b84bf\") " pod="opendatahub/maas-controller-6565584cfb-4hgzm"
Oct 30 21:01:28.848439 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:28.848272    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d799739f-553c-4245-8ee7-4742ba6b84bf-webhook-cert\") pod \"maas-controller-6565584cfb-4hgzm\" (UID: \"d799739f-553c-4245-8ee7-4742ba6b84bf\") " pod="opendatahub/maas-controller-6565584cfb-4hgzm"
Oct 30 21:01:28.848439 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:28.848306    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mvgm\" (UniqueName: \"kubernetes.io/projected/d799739f-553c-4245-8ee7-4742ba6b84bf-kube-api-access-2mvgm\") pod \"maas-controller-6565584cfb-4hgzm\" (UID: \"d799739f-553c-4245-8ee7-4742ba6b84bf\") " pod="opendatahub/maas-controller-6565584cfb-4hgzm"
Oct 30 21:01:28.857343 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:28.857265    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d799739f-553c-4245-8ee7-4742ba6b84bf-metrics-certs\") pod \"maas-controller-6565584cfb-4hgzm\" (UID: \"d799739f-553c-4245-8ee7-4742ba6b84bf\") " pod="opendatahub/maas-controller-6565584cfb-4hgzm"
Oct 30 21:01:28.858094 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:28.858063    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d799739f-553c-4245-8ee7-4742ba6b84bf-webhook-cert\") pod \"maas-controller-6565584cfb-4hgzm\" (UID: \"d799739f-553c-4245-8ee7-4742ba6b84bf\") " pod="opendatahub/maas-controller-6565584cfb-4hgzm"
Oct 30 21:01:28.911913 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:28.908502    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mvgm\" (UniqueName: \"kubernetes.io/projected/d799739f-553c-4245-8ee7-4742ba6b84bf-kube-api-access-2mvgm\") pod \"maas-controller-6565584cfb-4hgzm\" (UID: \"d799739f-553c-4245-8ee7-4742ba6b84bf\") " pod="opendatahub/maas-controller-6565584cfb-4hgzm"
Oct 30 21:01:28.943345 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:28.943035    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6565584cfb-4hgzm"
Oct 30 21:01:29.451848 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:29.447549    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6565584cfb-4hgzm"]
Oct 30 21:01:29.586202 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:29.586115    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6565584cfb-4hgzm" event={"ID":"d799739f-553c-4245-8ee7-4742ba6b84bf","Type":"ContainerStarted","Data":"3a4fef64f7471ccce2a18f98449cbc661eab8fb39ac2b7f6416f4d268eef9598"}
Oct 30 21:01:31.604765 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:31.604726    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6565584cfb-4hgzm" event={"ID":"d799739f-553c-4245-8ee7-4742ba6b84bf","Type":"ContainerStarted","Data":"a270f260b28510ac538f85ee909b9d7589f67acb5d9669bf5da68adfc567bcbe"}
Oct 30 21:01:31.605251 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:31.604877    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6565584cfb-4hgzm"
Oct 30 21:01:31.727241 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:31.727191    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6565584cfb-4hgzm" podStartSLOduration=2.7971442509999997 podStartE2EDuration="3.727177335s" totalImagesPullingTime="930.033084ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:31.724741371 +0000 UTC m=+2103.641762908" watchObservedRunningTime="2026-10-30 21:01:31.727177335 +0000 UTC m=+2103.644198870"
Oct 30 21:01:32.610624 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:32.610588    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:32.777752 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:32.777714    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:34.589273 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:34.589237    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:36.061544 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:36.061497    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:41.212880 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:41.212801    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f"]
Oct 30 21:01:41.213283 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:41.213239    5071 cpu_manager.go:414] "containerMap: removing container" podUID="b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607" containerName="manager"
Oct 30 21:01:41.213283 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:41.213250    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2ecc2aa-5d8a-400a-b7c4-e4d55be1e607" containerName="manager"
Oct 30 21:01:41.218319 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:41.218303    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f"
Oct 30 21:01:41.303745 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:41.303709    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/562c4391-03f9-470d-a007-b4931bd3f421-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xhg2f\" (UID: \"562c4391-03f9-470d-a007-b4931bd3f421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f"
Oct 30 21:01:41.303745 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:41.303749    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmnh8\" (UniqueName: \"kubernetes.io/projected/562c4391-03f9-470d-a007-b4931bd3f421-kube-api-access-tmnh8\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xhg2f\" (UID: \"562c4391-03f9-470d-a007-b4931bd3f421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f"
Oct 30 21:01:41.303989 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:41.303772    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/562c4391-03f9-470d-a007-b4931bd3f421-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xhg2f\" (UID: \"562c4391-03f9-470d-a007-b4931bd3f421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f"
Oct 30 21:01:41.303989 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:41.303908    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/562c4391-03f9-470d-a007-b4931bd3f421-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xhg2f\" (UID: \"562c4391-03f9-470d-a007-b4931bd3f421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f"
Oct 30 21:01:41.303989 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:41.303954    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/562c4391-03f9-470d-a007-b4931bd3f421-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xhg2f\" (UID: \"562c4391-03f9-470d-a007-b4931bd3f421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f"
Oct 30 21:01:41.303989 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:41.303980    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/562c4391-03f9-470d-a007-b4931bd3f421-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xhg2f\" (UID: \"562c4391-03f9-470d-a007-b4931bd3f421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f"
Oct 30 21:01:41.304211 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:41.304018    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/562c4391-03f9-470d-a007-b4931bd3f421-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xhg2f\" (UID: \"562c4391-03f9-470d-a007-b4931bd3f421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f"
Oct 30 21:01:41.304211 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:41.304116    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/562c4391-03f9-470d-a007-b4931bd3f421-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xhg2f\" (UID: \"562c4391-03f9-470d-a007-b4931bd3f421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f"
Oct 30 21:01:41.304211 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:41.304177    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/562c4391-03f9-470d-a007-b4931bd3f421-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xhg2f\" (UID: \"562c4391-03f9-470d-a007-b4931bd3f421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f"
Oct 30 21:01:41.405255 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:41.405224    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/562c4391-03f9-470d-a007-b4931bd3f421-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xhg2f\" (UID: \"562c4391-03f9-470d-a007-b4931bd3f421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f"
Oct 30 21:01:41.405431 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:41.405276    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/562c4391-03f9-470d-a007-b4931bd3f421-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xhg2f\" (UID: \"562c4391-03f9-470d-a007-b4931bd3f421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f"
Oct 30 21:01:41.405431 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:41.405299    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/562c4391-03f9-470d-a007-b4931bd3f421-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xhg2f\" (UID: \"562c4391-03f9-470d-a007-b4931bd3f421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f"
Oct 30 21:01:41.405431 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:41.405339    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/562c4391-03f9-470d-a007-b4931bd3f421-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xhg2f\" (UID: \"562c4391-03f9-470d-a007-b4931bd3f421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f"
Oct 30 21:01:41.405431 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:41.405365    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmnh8\" (UniqueName: \"kubernetes.io/projected/562c4391-03f9-470d-a007-b4931bd3f421-kube-api-access-tmnh8\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xhg2f\" (UID: \"562c4391-03f9-470d-a007-b4931bd3f421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f"
Oct 30 21:01:41.405431 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:41.405396    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/562c4391-03f9-470d-a007-b4931bd3f421-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xhg2f\" (UID: \"562c4391-03f9-470d-a007-b4931bd3f421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f"
Oct 30 21:01:41.405714 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:41.405435    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/562c4391-03f9-470d-a007-b4931bd3f421-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xhg2f\" (UID: \"562c4391-03f9-470d-a007-b4931bd3f421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f"
Oct 30 21:01:41.405714 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:41.405469    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/562c4391-03f9-470d-a007-b4931bd3f421-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xhg2f\" (UID: \"562c4391-03f9-470d-a007-b4931bd3f421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f"
Oct 30 21:01:41.405714 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:41.405503    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/562c4391-03f9-470d-a007-b4931bd3f421-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xhg2f\" (UID: \"562c4391-03f9-470d-a007-b4931bd3f421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f"
Oct 30 21:01:41.405714 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:41.405694    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/562c4391-03f9-470d-a007-b4931bd3f421-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xhg2f\" (UID: \"562c4391-03f9-470d-a007-b4931bd3f421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f"
Oct 30 21:01:41.405988 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:41.405971    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/562c4391-03f9-470d-a007-b4931bd3f421-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xhg2f\" (UID: \"562c4391-03f9-470d-a007-b4931bd3f421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f"
Oct 30 21:01:41.406062 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:41.406001    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/562c4391-03f9-470d-a007-b4931bd3f421-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xhg2f\" (UID: \"562c4391-03f9-470d-a007-b4931bd3f421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f"
Oct 30 21:01:41.406155 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:41.406127    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/562c4391-03f9-470d-a007-b4931bd3f421-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xhg2f\" (UID: \"562c4391-03f9-470d-a007-b4931bd3f421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f"
Oct 30 21:01:41.406229 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:41.406187    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/562c4391-03f9-470d-a007-b4931bd3f421-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xhg2f\" (UID: \"562c4391-03f9-470d-a007-b4931bd3f421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f"
Oct 30 21:01:41.408028 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:41.408007    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/562c4391-03f9-470d-a007-b4931bd3f421-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xhg2f\" (UID: \"562c4391-03f9-470d-a007-b4931bd3f421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f"
Oct 30 21:01:41.408176 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:41.408160    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/562c4391-03f9-470d-a007-b4931bd3f421-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xhg2f\" (UID: \"562c4391-03f9-470d-a007-b4931bd3f421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f"
Oct 30 21:01:41.420648 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:41.420604    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f"]
Oct 30 21:01:41.463880 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:41.463793    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/562c4391-03f9-470d-a007-b4931bd3f421-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xhg2f\" (UID: \"562c4391-03f9-470d-a007-b4931bd3f421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f"
Oct 30 21:01:41.464021 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:41.463933    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmnh8\" (UniqueName: \"kubernetes.io/projected/562c4391-03f9-470d-a007-b4931bd3f421-kube-api-access-tmnh8\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xhg2f\" (UID: \"562c4391-03f9-470d-a007-b4931bd3f421\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f"
Oct 30 21:01:41.531737 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:41.531701    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f"
Oct 30 21:01:41.961153 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:41.961126    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f"]
Oct 30 21:01:41.963386 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:01:41.963349    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod562c4391_03f9_470d_a007_b4931bd3f421.slice/crio-eb03f6f03985aa99eb11758328b66af4fab755d0b83529ebae821e247fe1fdba WatchSource:0}: Error finding container eb03f6f03985aa99eb11758328b66af4fab755d0b83529ebae821e247fe1fdba: Status 404 returned error can't find the container with id eb03f6f03985aa99eb11758328b66af4fab755d0b83529ebae821e247fe1fdba
Oct 30 21:01:41.965572 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:41.965538    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:01:41.965656 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:41.965610    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:01:41.965656 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:41.965647    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:01:42.541033 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:42.540944    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:42.612888 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:42.612856    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6565584cfb-4hgzm"
Oct 30 21:01:42.664142 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:42.664102    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f" event={"ID":"562c4391-03f9-470d-a007-b4931bd3f421","Type":"ContainerStarted","Data":"cc08a5edb1e8dbbeccc06e9f6fbcdb64119d0e45dd6dc95c6ea5200ac177280b"}
Oct 30 21:01:42.664357 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:42.664175    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f" event={"ID":"562c4391-03f9-470d-a007-b4931bd3f421","Type":"ContainerStarted","Data":"eb03f6f03985aa99eb11758328b66af4fab755d0b83529ebae821e247fe1fdba"}
Oct 30 21:01:43.549847 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:43.546800    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f"
Oct 30 21:01:43.551996 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:43.551963    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-xhg2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Oct 30 21:01:43.552245 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:43.552199    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f" podUID="562c4391-03f9-470d-a007-b4931bd3f421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Oct 30 21:01:44.533175 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:44.533140    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-xhg2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Oct 30 21:01:44.533471 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:44.533439    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f" podUID="562c4391-03f9-470d-a007-b4931bd3f421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Oct 30 21:01:45.540847 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:45.537234    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-xhg2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Oct 30 21:01:45.540847 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:45.537299    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f" podUID="562c4391-03f9-470d-a007-b4931bd3f421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Oct 30 21:01:46.532289 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:46.532247    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-xhg2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Oct 30 21:01:46.532628 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:46.532595    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f" podUID="562c4391-03f9-470d-a007-b4931bd3f421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Oct 30 21:01:46.727458 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:01:46.727413    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:01:47.544856 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:47.540202    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-xhg2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Oct 30 21:01:47.544856 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:47.540267    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f" podUID="562c4391-03f9-470d-a007-b4931bd3f421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Oct 30 21:01:48.532782 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:48.532744    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-xhg2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Oct 30 21:01:48.533235 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:48.532806    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f" podUID="562c4391-03f9-470d-a007-b4931bd3f421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Oct 30 21:01:49.532226 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:49.532189    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-xhg2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Oct 30 21:01:49.532455 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:49.532243    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f" podUID="562c4391-03f9-470d-a007-b4931bd3f421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Oct 30 21:01:50.067105 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:50.067032    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f" podStartSLOduration=9.067011027 podStartE2EDuration="9.067011027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:42.853536128 +0000 UTC m=+2114.770557672" watchObservedRunningTime="2026-10-30 21:01:50.067011027 +0000 UTC m=+2121.984032564"
Oct 30 21:01:50.069607 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:50.069581    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jwpzr"]
Oct 30 21:01:50.074005 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:50.073977    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jwpzr"
Oct 30 21:01:50.101400 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:50.101366    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e53d3b25-1074-434c-a332-00a3b737f173-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-jwpzr\" (UID: \"e53d3b25-1074-434c-a332-00a3b737f173\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jwpzr"
Oct 30 21:01:50.101561 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:50.101471    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xtdz\" (UniqueName: \"kubernetes.io/projected/e53d3b25-1074-434c-a332-00a3b737f173-kube-api-access-9xtdz\") pod \"maas-api-revoke-sub-85237e45-jwpzr\" (UID: \"e53d3b25-1074-434c-a332-00a3b737f173\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jwpzr"
Oct 30 21:01:50.101561 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:50.101522    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e53d3b25-1074-434c-a332-00a3b737f173-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-jwpzr\" (UID: \"e53d3b25-1074-434c-a332-00a3b737f173\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jwpzr"
Oct 30 21:01:50.133133 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:50.130798    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jwpzr"]
Oct 30 21:01:50.176725 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:50.176690    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:01:50.176725 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:50.176691    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:01:50.202539 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:50.202508    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xtdz\" (UniqueName: \"kubernetes.io/projected/e53d3b25-1074-434c-a332-00a3b737f173-kube-api-access-9xtdz\") pod \"maas-api-revoke-sub-85237e45-jwpzr\" (UID: \"e53d3b25-1074-434c-a332-00a3b737f173\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jwpzr"
Oct 30 21:01:50.202753 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:50.202559    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e53d3b25-1074-434c-a332-00a3b737f173-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-jwpzr\" (UID: \"e53d3b25-1074-434c-a332-00a3b737f173\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jwpzr"
Oct 30 21:01:50.202753 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:50.202629    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e53d3b25-1074-434c-a332-00a3b737f173-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-jwpzr\" (UID: \"e53d3b25-1074-434c-a332-00a3b737f173\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jwpzr"
Oct 30 21:01:50.203303 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:50.203280    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e53d3b25-1074-434c-a332-00a3b737f173-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-jwpzr\" (UID: \"e53d3b25-1074-434c-a332-00a3b737f173\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jwpzr"
Oct 30 21:01:50.221265 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:50.221237    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:01:50.225774 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:50.225749    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e53d3b25-1074-434c-a332-00a3b737f173-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-jwpzr\" (UID: \"e53d3b25-1074-434c-a332-00a3b737f173\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jwpzr"
Oct 30 21:01:50.226343 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:50.226323    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 21:01:50.261335 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:50.261292    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xtdz\" (UniqueName: \"kubernetes.io/projected/e53d3b25-1074-434c-a332-00a3b737f173-kube-api-access-9xtdz\") pod \"maas-api-revoke-sub-85237e45-jwpzr\" (UID: \"e53d3b25-1074-434c-a332-00a3b737f173\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jwpzr"
Oct 30 21:01:50.390878 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:50.390751    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jwpzr"
Oct 30 21:01:50.598218 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:50.598177    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f"
Oct 30 21:01:50.599026 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:50.598986    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f"
Oct 30 21:01:50.599812 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:50.599794    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f"
Oct 30 21:01:50.777917 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:01:50.777882    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode53d3b25_1074_434c_a332_00a3b737f173.slice/crio-d37c16f28ac1adb24ae6147e2b8f94c128bf334269b874f6ebb21014974f6f5d WatchSource:0}: Error finding container d37c16f28ac1adb24ae6147e2b8f94c128bf334269b874f6ebb21014974f6f5d: Status 404 returned error can't find the container with id d37c16f28ac1adb24ae6147e2b8f94c128bf334269b874f6ebb21014974f6f5d
Oct 30 21:01:50.778131 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:50.778107    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jwpzr"]
Oct 30 21:01:51.718395 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:51.718360    5071 generic.go:356] "Generic (PLEG): container finished" podID="e53d3b25-1074-434c-a332-00a3b737f173" containerID="83a374a482d0674209c062d1052135eaeb5b3f370c4bea3ad745a1085cd1bb31" exitCode=0
Oct 30 21:01:51.718796 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:51.718448    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jwpzr" event={"ID":"e53d3b25-1074-434c-a332-00a3b737f173","Type":"ContainerDied","Data":"83a374a482d0674209c062d1052135eaeb5b3f370c4bea3ad745a1085cd1bb31"}
Oct 30 21:01:51.718796 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:51.718502    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jwpzr" event={"ID":"e53d3b25-1074-434c-a332-00a3b737f173","Type":"ContainerStarted","Data":"d37c16f28ac1adb24ae6147e2b8f94c128bf334269b874f6ebb21014974f6f5d"}
Oct 30 21:01:52.884950 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:52.884923    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jwpzr"
Oct 30 21:01:52.929377 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:52.929351    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e53d3b25-1074-434c-a332-00a3b737f173-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e53d3b25-1074-434c-a332-00a3b737f173-maas-api-openshift-ca\") pod \"e53d3b25-1074-434c-a332-00a3b737f173\" (UID: \"e53d3b25-1074-434c-a332-00a3b737f173\") "
Oct 30 21:01:52.929539 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:52.929456    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e53d3b25-1074-434c-a332-00a3b737f173-kube-api-access-9xtdz\" (UniqueName: \"kubernetes.io/projected/e53d3b25-1074-434c-a332-00a3b737f173-kube-api-access-9xtdz\") pod \"e53d3b25-1074-434c-a332-00a3b737f173\" (UID: \"e53d3b25-1074-434c-a332-00a3b737f173\") "
Oct 30 21:01:52.929539 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:52.929525    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e53d3b25-1074-434c-a332-00a3b737f173-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e53d3b25-1074-434c-a332-00a3b737f173-maas-api-xks-ca\") pod \"e53d3b25-1074-434c-a332-00a3b737f173\" (UID: \"e53d3b25-1074-434c-a332-00a3b737f173\") "
Oct 30 21:01:52.929698 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:52.929675    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e53d3b25-1074-434c-a332-00a3b737f173-maas-api-openshift-ca" pod "e53d3b25-1074-434c-a332-00a3b737f173" (UID: "e53d3b25-1074-434c-a332-00a3b737f173"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:52.929941 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:52.929923    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e53d3b25-1074-434c-a332-00a3b737f173-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:52.931757 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:52.931729    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e53d3b25-1074-434c-a332-00a3b737f173-maas-api-xks-ca" pod "e53d3b25-1074-434c-a332-00a3b737f173" (UID: "e53d3b25-1074-434c-a332-00a3b737f173"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:52.931883 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:52.931768    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e53d3b25-1074-434c-a332-00a3b737f173-kube-api-access-9xtdz" pod "e53d3b25-1074-434c-a332-00a3b737f173" (UID: "e53d3b25-1074-434c-a332-00a3b737f173"). InnerVolumeSpecName "kube-api-access-9xtdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:53.030850 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:53.030741    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xtdz\" (UniqueName: \"kubernetes.io/projected/e53d3b25-1074-434c-a332-00a3b737f173-kube-api-access-9xtdz\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:53.030850 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:53.030779    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e53d3b25-1074-434c-a332-00a3b737f173-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:53.739378 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:53.739341    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jwpzr"
Oct 30 21:01:53.739572 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:53.739370    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jwpzr" event={"ID":"e53d3b25-1074-434c-a332-00a3b737f173","Type":"ContainerDied","Data":"d37c16f28ac1adb24ae6147e2b8f94c128bf334269b874f6ebb21014974f6f5d"}
Oct 30 21:01:53.739572 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:53.739459    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d37c16f28ac1adb24ae6147e2b8f94c128bf334269b874f6ebb21014974f6f5d"
Oct 30 21:01:55.450559 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:01:55.450526    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:02:01.156810 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:01.156763    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:01.331682 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:01.331640    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:02.535425 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:02.535367    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:02.662972 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:02.662933    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6565584cfb-4hgzm"]
Oct 30 21:02:02.663261 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:02.663203    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6565584cfb-4hgzm" podUID="d799739f-553c-4245-8ee7-4742ba6b84bf" containerName="manager" containerID="cri-o://a270f260b28510ac538f85ee909b9d7589f67acb5d9669bf5da68adfc567bcbe" gracePeriod=30
Oct 30 21:02:02.798337 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:02.798292    5071 generic.go:356] "Generic (PLEG): container finished" podID="d799739f-553c-4245-8ee7-4742ba6b84bf" containerID="a270f260b28510ac538f85ee909b9d7589f67acb5d9669bf5da68adfc567bcbe" exitCode=0
Oct 30 21:02:02.798490 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:02.798359    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6565584cfb-4hgzm" event={"ID":"d799739f-553c-4245-8ee7-4742ba6b84bf","Type":"ContainerDied","Data":"a270f260b28510ac538f85ee909b9d7589f67acb5d9669bf5da68adfc567bcbe"}
Oct 30 21:02:02.942980 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:02.942954    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6565584cfb-4hgzm"
Oct 30 21:02:03.040221 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:03.040185    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d799739f-553c-4245-8ee7-4742ba6b84bf-kube-api-access-2mvgm\" (UniqueName: \"kubernetes.io/projected/d799739f-553c-4245-8ee7-4742ba6b84bf-kube-api-access-2mvgm\") pod \"d799739f-553c-4245-8ee7-4742ba6b84bf\" (UID: \"d799739f-553c-4245-8ee7-4742ba6b84bf\") "
Oct 30 21:02:03.040397 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:03.040269    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d799739f-553c-4245-8ee7-4742ba6b84bf-webhook-cert\" (UniqueName: \"kubernetes.io/secret/d799739f-553c-4245-8ee7-4742ba6b84bf-webhook-cert\") pod \"d799739f-553c-4245-8ee7-4742ba6b84bf\" (UID: \"d799739f-553c-4245-8ee7-4742ba6b84bf\") "
Oct 30 21:02:03.040397 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:03.040288    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d799739f-553c-4245-8ee7-4742ba6b84bf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d799739f-553c-4245-8ee7-4742ba6b84bf-metrics-certs\") pod \"d799739f-553c-4245-8ee7-4742ba6b84bf\" (UID: \"d799739f-553c-4245-8ee7-4742ba6b84bf\") "
Oct 30 21:02:03.042517 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:03.042479    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d799739f-553c-4245-8ee7-4742ba6b84bf-kube-api-access-2mvgm" pod "d799739f-553c-4245-8ee7-4742ba6b84bf" (UID: "d799739f-553c-4245-8ee7-4742ba6b84bf"). InnerVolumeSpecName "kube-api-access-2mvgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:03.042658 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:03.042593    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d799739f-553c-4245-8ee7-4742ba6b84bf-webhook-cert" pod "d799739f-553c-4245-8ee7-4742ba6b84bf" (UID: "d799739f-553c-4245-8ee7-4742ba6b84bf"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:03.042658 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:03.042615    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d799739f-553c-4245-8ee7-4742ba6b84bf-metrics-certs" pod "d799739f-553c-4245-8ee7-4742ba6b84bf" (UID: "d799739f-553c-4245-8ee7-4742ba6b84bf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:03.142177 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:03.142135    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mvgm\" (UniqueName: \"kubernetes.io/projected/d799739f-553c-4245-8ee7-4742ba6b84bf-kube-api-access-2mvgm\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:02:03.142177 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:03.142176    5071 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d799739f-553c-4245-8ee7-4742ba6b84bf-webhook-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:02:03.142392 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:03.142194    5071 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d799739f-553c-4245-8ee7-4742ba6b84bf-metrics-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:02:03.805130 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:03.805101    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6565584cfb-4hgzm"
Oct 30 21:02:03.805616 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:03.805127    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6565584cfb-4hgzm" event={"ID":"d799739f-553c-4245-8ee7-4742ba6b84bf","Type":"ContainerDied","Data":"3a4fef64f7471ccce2a18f98449cbc661eab8fb39ac2b7f6416f4d268eef9598"}
Oct 30 21:02:03.805616 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:03.805190    5071 scope.go:122] "RemoveContainer" containerID="a270f260b28510ac538f85ee909b9d7589f67acb5d9669bf5da68adfc567bcbe"
Oct 30 21:02:03.973228 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:03.973196    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6565584cfb-4hgzm"]
Oct 30 21:02:04.023010 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:04.022976    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6565584cfb-4hgzm"]
Oct 30 21:02:04.200877 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:04.200771    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d799739f-553c-4245-8ee7-4742ba6b84bf" path="/var/lib/kubelet/pods/d799739f-553c-4245-8ee7-4742ba6b84bf/volumes"
Oct 30 21:02:05.320432 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:05.320394    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:05.447514 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:05.447482    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6565584cfb-zrltv"]
Oct 30 21:02:05.447925 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:05.447912    5071 cpu_manager.go:414] "containerMap: removing container" podUID="d799739f-553c-4245-8ee7-4742ba6b84bf" containerName="manager"
Oct 30 21:02:05.447972 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:05.447926    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d799739f-553c-4245-8ee7-4742ba6b84bf" containerName="manager"
Oct 30 21:02:05.447972 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:05.447953    5071 cpu_manager.go:414] "containerMap: removing container" podUID="e53d3b25-1074-434c-a332-00a3b737f173" containerName="revoke-keys"
Oct 30 21:02:05.447972 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:05.447958    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e53d3b25-1074-434c-a332-00a3b737f173" containerName="revoke-keys"
Oct 30 21:02:05.448077 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:05.448033    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="d799739f-553c-4245-8ee7-4742ba6b84bf" containerName="manager"
Oct 30 21:02:05.448077 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:05.448046    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="e53d3b25-1074-434c-a332-00a3b737f173" containerName="revoke-keys"
Oct 30 21:02:05.452414 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:05.452396    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6565584cfb-zrltv"
Oct 30 21:02:05.555132 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:05.555098    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 21:02:05.555132 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:05.555125    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sk69l\""
Oct 30 21:02:05.555408 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:05.555143    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 21:02:05.555408 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:05.555125    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 21:02:05.558282 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:05.558257    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6565584cfb-zrltv"]
Oct 30 21:02:05.564272 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:05.564253    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kbdg\" (UniqueName: \"kubernetes.io/projected/fe81a376-1de8-4c13-8d8a-49c638d37d00-kube-api-access-2kbdg\") pod \"maas-controller-6565584cfb-zrltv\" (UID: \"fe81a376-1de8-4c13-8d8a-49c638d37d00\") " pod="opendatahub/maas-controller-6565584cfb-zrltv"
Oct 30 21:02:05.564411 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:05.564296    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe81a376-1de8-4c13-8d8a-49c638d37d00-metrics-certs\") pod \"maas-controller-6565584cfb-zrltv\" (UID: \"fe81a376-1de8-4c13-8d8a-49c638d37d00\") " pod="opendatahub/maas-controller-6565584cfb-zrltv"
Oct 30 21:02:05.564411 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:05.564317    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe81a376-1de8-4c13-8d8a-49c638d37d00-webhook-cert\") pod \"maas-controller-6565584cfb-zrltv\" (UID: \"fe81a376-1de8-4c13-8d8a-49c638d37d00\") " pod="opendatahub/maas-controller-6565584cfb-zrltv"
Oct 30 21:02:05.664982 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:05.664900    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kbdg\" (UniqueName: \"kubernetes.io/projected/fe81a376-1de8-4c13-8d8a-49c638d37d00-kube-api-access-2kbdg\") pod \"maas-controller-6565584cfb-zrltv\" (UID: \"fe81a376-1de8-4c13-8d8a-49c638d37d00\") " pod="opendatahub/maas-controller-6565584cfb-zrltv"
Oct 30 21:02:05.664982 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:05.664968    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe81a376-1de8-4c13-8d8a-49c638d37d00-metrics-certs\") pod \"maas-controller-6565584cfb-zrltv\" (UID: \"fe81a376-1de8-4c13-8d8a-49c638d37d00\") " pod="opendatahub/maas-controller-6565584cfb-zrltv"
Oct 30 21:02:05.665266 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:05.665002    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe81a376-1de8-4c13-8d8a-49c638d37d00-webhook-cert\") pod \"maas-controller-6565584cfb-zrltv\" (UID: \"fe81a376-1de8-4c13-8d8a-49c638d37d00\") " pod="opendatahub/maas-controller-6565584cfb-zrltv"
Oct 30 21:02:05.667848 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:05.667787    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe81a376-1de8-4c13-8d8a-49c638d37d00-webhook-cert\") pod \"maas-controller-6565584cfb-zrltv\" (UID: \"fe81a376-1de8-4c13-8d8a-49c638d37d00\") " pod="opendatahub/maas-controller-6565584cfb-zrltv"
Oct 30 21:02:05.667848 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:05.667803    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe81a376-1de8-4c13-8d8a-49c638d37d00-metrics-certs\") pod \"maas-controller-6565584cfb-zrltv\" (UID: \"fe81a376-1de8-4c13-8d8a-49c638d37d00\") " pod="opendatahub/maas-controller-6565584cfb-zrltv"
Oct 30 21:02:05.724477 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:05.724445    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kbdg\" (UniqueName: \"kubernetes.io/projected/fe81a376-1de8-4c13-8d8a-49c638d37d00-kube-api-access-2kbdg\") pod \"maas-controller-6565584cfb-zrltv\" (UID: \"fe81a376-1de8-4c13-8d8a-49c638d37d00\") " pod="opendatahub/maas-controller-6565584cfb-zrltv"
Oct 30 21:02:05.765955 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:05.765922    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6565584cfb-zrltv"
Oct 30 21:02:06.147327 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:06.147303    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6565584cfb-zrltv"]
Oct 30 21:02:06.149135 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:02:06.149102    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe81a376_1de8_4c13_8d8a_49c638d37d00.slice/crio-59ea399c1238d26368b22556dfd75c1a325badb7e97d053d983c7d8cdf913662 WatchSource:0}: Error finding container 59ea399c1238d26368b22556dfd75c1a325badb7e97d053d983c7d8cdf913662: Status 404 returned error can't find the container with id 59ea399c1238d26368b22556dfd75c1a325badb7e97d053d983c7d8cdf913662
Oct 30 21:02:06.824360 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:06.824323    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6565584cfb-zrltv" event={"ID":"fe81a376-1de8-4c13-8d8a-49c638d37d00","Type":"ContainerStarted","Data":"59ea399c1238d26368b22556dfd75c1a325badb7e97d053d983c7d8cdf913662"}
Oct 30 21:02:07.830459 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:07.830424    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6565584cfb-zrltv" event={"ID":"fe81a376-1de8-4c13-8d8a-49c638d37d00","Type":"ContainerStarted","Data":"12b3a7dea3b5ceb051d6e254ebaf0bf62e031f25f5b84b374a3cacc471fdf27d"}
Oct 30 21:02:07.830916 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:07.830551    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6565584cfb-zrltv"
Oct 30 21:02:07.949040 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:07.948974    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6565584cfb-zrltv" podStartSLOduration=2.111950512 podStartE2EDuration="2.948954388s" totalImagesPullingTime="837.003876ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:07.947366858 +0000 UTC m=+2139.864388412" watchObservedRunningTime="2026-10-30 21:02:07.948954388 +0000 UTC m=+2139.865975937"
Oct 30 21:02:16.728096 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:02:16.728059    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:02:18.838452 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:18.838409    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6565584cfb-zrltv"
Oct 30 21:02:21.868017 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:21.867972    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:26.051192 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:26.051155    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:26.208850 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:26.208773    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:34.920076 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:34.920037    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:37.451552 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:37.451514    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:38.661451 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:38.661417    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:38.828572 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:38.828539    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:42.882534 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:42.882489    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:02:46.727537 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:02:46.727495    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:02:52.163739 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:02:52.163694    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:03:02.565389 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:02.565355    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xz7s7"]
Oct 30 21:03:02.570569 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:02.570543    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xz7s7"
Oct 30 21:03:02.627054 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:02.627022    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xz7s7"]
Oct 30 21:03:02.673809 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:02.673767    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:03:02.673809 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:02.673790    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:03:02.673809 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:02.673791    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:03:02.674063 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:02.673878    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 21:03:02.684360 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:02.684323    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2be78af-3d58-44af-afe4-0a684ed7a7e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-xz7s7\" (UID: \"a2be78af-3d58-44af-afe4-0a684ed7a7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xz7s7"
Oct 30 21:03:02.684531 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:02.684460    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncnhf\" (UniqueName: \"kubernetes.io/projected/a2be78af-3d58-44af-afe4-0a684ed7a7e5-kube-api-access-ncnhf\") pod \"maas-api-revoke-sub-c1e271a9-xz7s7\" (UID: \"a2be78af-3d58-44af-afe4-0a684ed7a7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xz7s7"
Oct 30 21:03:02.684674 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:02.684653    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2be78af-3d58-44af-afe4-0a684ed7a7e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-xz7s7\" (UID: \"a2be78af-3d58-44af-afe4-0a684ed7a7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xz7s7"
Oct 30 21:03:02.785526 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:02.785491    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2be78af-3d58-44af-afe4-0a684ed7a7e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-xz7s7\" (UID: \"a2be78af-3d58-44af-afe4-0a684ed7a7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xz7s7"
Oct 30 21:03:02.785709 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:02.785547    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncnhf\" (UniqueName: \"kubernetes.io/projected/a2be78af-3d58-44af-afe4-0a684ed7a7e5-kube-api-access-ncnhf\") pod \"maas-api-revoke-sub-c1e271a9-xz7s7\" (UID: \"a2be78af-3d58-44af-afe4-0a684ed7a7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xz7s7"
Oct 30 21:03:02.785709 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:02.785590    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2be78af-3d58-44af-afe4-0a684ed7a7e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-xz7s7\" (UID: \"a2be78af-3d58-44af-afe4-0a684ed7a7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xz7s7"
Oct 30 21:03:02.786273 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:02.786243    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2be78af-3d58-44af-afe4-0a684ed7a7e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-xz7s7\" (UID: \"a2be78af-3d58-44af-afe4-0a684ed7a7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xz7s7"
Oct 30 21:03:02.788049 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:02.788031    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2be78af-3d58-44af-afe4-0a684ed7a7e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-xz7s7\" (UID: \"a2be78af-3d58-44af-afe4-0a684ed7a7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xz7s7"
Oct 30 21:03:02.844046 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:02.843976    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncnhf\" (UniqueName: \"kubernetes.io/projected/a2be78af-3d58-44af-afe4-0a684ed7a7e5-kube-api-access-ncnhf\") pod \"maas-api-revoke-sub-c1e271a9-xz7s7\" (UID: \"a2be78af-3d58-44af-afe4-0a684ed7a7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xz7s7"
Oct 30 21:03:02.882986 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:02.882954    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xz7s7"
Oct 30 21:03:03.260730 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:03.260699    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xz7s7"]
Oct 30 21:03:03.262657 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:03:03.262622    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2be78af_3d58_44af_afe4_0a684ed7a7e5.slice/crio-29fb6f4470b92fd74ef861b0d8e48abcf542ef213e8f5101dfa54a4fec7827bd WatchSource:0}: Error finding container 29fb6f4470b92fd74ef861b0d8e48abcf542ef213e8f5101dfa54a4fec7827bd: Status 404 returned error can't find the container with id 29fb6f4470b92fd74ef861b0d8e48abcf542ef213e8f5101dfa54a4fec7827bd
Oct 30 21:03:04.161837 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:04.161793    5071 generic.go:356] "Generic (PLEG): container finished" podID="a2be78af-3d58-44af-afe4-0a684ed7a7e5" containerID="d3832bd9534764898de630ab20b3194da5306a10ecda73bb9aed983754587699" exitCode=0
Oct 30 21:03:04.162214 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:04.161863    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xz7s7" event={"ID":"a2be78af-3d58-44af-afe4-0a684ed7a7e5","Type":"ContainerDied","Data":"d3832bd9534764898de630ab20b3194da5306a10ecda73bb9aed983754587699"}
Oct 30 21:03:04.162214 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:04.161927    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xz7s7" event={"ID":"a2be78af-3d58-44af-afe4-0a684ed7a7e5","Type":"ContainerStarted","Data":"29fb6f4470b92fd74ef861b0d8e48abcf542ef213e8f5101dfa54a4fec7827bd"}
Oct 30 21:03:04.430091 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:04.430006    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:04.588769 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:04.588729    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:05.325456 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:05.325427    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xz7s7"
Oct 30 21:03:05.509958 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:05.509868    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2be78af-3d58-44af-afe4-0a684ed7a7e5-kube-api-access-ncnhf\" (UniqueName: \"kubernetes.io/projected/a2be78af-3d58-44af-afe4-0a684ed7a7e5-kube-api-access-ncnhf\") pod \"a2be78af-3d58-44af-afe4-0a684ed7a7e5\" (UID: \"a2be78af-3d58-44af-afe4-0a684ed7a7e5\") "
Oct 30 21:03:05.509958 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:05.509928    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2be78af-3d58-44af-afe4-0a684ed7a7e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2be78af-3d58-44af-afe4-0a684ed7a7e5-maas-api-openshift-ca\") pod \"a2be78af-3d58-44af-afe4-0a684ed7a7e5\" (UID: \"a2be78af-3d58-44af-afe4-0a684ed7a7e5\") "
Oct 30 21:03:05.510205 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:05.509996    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2be78af-3d58-44af-afe4-0a684ed7a7e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2be78af-3d58-44af-afe4-0a684ed7a7e5-maas-api-xks-ca\") pod \"a2be78af-3d58-44af-afe4-0a684ed7a7e5\" (UID: \"a2be78af-3d58-44af-afe4-0a684ed7a7e5\") "
Oct 30 21:03:05.510355 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:05.510328    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2be78af-3d58-44af-afe4-0a684ed7a7e5-maas-api-openshift-ca" pod "a2be78af-3d58-44af-afe4-0a684ed7a7e5" (UID: "a2be78af-3d58-44af-afe4-0a684ed7a7e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:05.512204 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:05.512179    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2be78af-3d58-44af-afe4-0a684ed7a7e5-maas-api-xks-ca" pod "a2be78af-3d58-44af-afe4-0a684ed7a7e5" (UID: "a2be78af-3d58-44af-afe4-0a684ed7a7e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:03:05.512302 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:05.512258    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2be78af-3d58-44af-afe4-0a684ed7a7e5-kube-api-access-ncnhf" pod "a2be78af-3d58-44af-afe4-0a684ed7a7e5" (UID: "a2be78af-3d58-44af-afe4-0a684ed7a7e5"). InnerVolumeSpecName "kube-api-access-ncnhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:05.611488 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:05.611454    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2be78af-3d58-44af-afe4-0a684ed7a7e5-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:05.611488 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:05.611485    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncnhf\" (UniqueName: \"kubernetes.io/projected/a2be78af-3d58-44af-afe4-0a684ed7a7e5-kube-api-access-ncnhf\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:05.611488 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:05.611496    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2be78af-3d58-44af-afe4-0a684ed7a7e5-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:06.172604 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:06.172569    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xz7s7"
Oct 30 21:03:06.172788 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:06.172604    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xz7s7" event={"ID":"a2be78af-3d58-44af-afe4-0a684ed7a7e5","Type":"ContainerDied","Data":"29fb6f4470b92fd74ef861b0d8e48abcf542ef213e8f5101dfa54a4fec7827bd"}
Oct 30 21:03:06.172788 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:06.172648    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29fb6f4470b92fd74ef861b0d8e48abcf542ef213e8f5101dfa54a4fec7827bd"
Oct 30 21:03:15.382729 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:15.382508    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:03:16.727048 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:03:16.727011    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:03:26.239602 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:26.239572    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:03:33.814969 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:33.814937    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hg58l"]
Oct 30 21:03:33.815387 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:33.815375    5071 cpu_manager.go:414] "containerMap: removing container" podUID="a2be78af-3d58-44af-afe4-0a684ed7a7e5" containerName="revoke-keys"
Oct 30 21:03:33.815434 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:33.815388    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2be78af-3d58-44af-afe4-0a684ed7a7e5" containerName="revoke-keys"
Oct 30 21:03:33.815471 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:33.815458    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2be78af-3d58-44af-afe4-0a684ed7a7e5" containerName="revoke-keys"
Oct 30 21:03:33.818882 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:33.818863    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hg58l"
Oct 30 21:03:33.876464 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:33.876428    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hg58l"]
Oct 30 21:03:33.922733 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:33.922698    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:03:33.922733 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:33.922715    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:03:33.922999 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:33.922792    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:03:33.922999 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:33.922885    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 21:03:33.930250 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:33.930218    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d35530f-1a3b-485d-a41f-dd20fbcb5b01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-hg58l\" (UID: \"2d35530f-1a3b-485d-a41f-dd20fbcb5b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hg58l"
Oct 30 21:03:33.930383 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:33.930267    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq2tr\" (UniqueName: \"kubernetes.io/projected/2d35530f-1a3b-485d-a41f-dd20fbcb5b01-kube-api-access-pq2tr\") pod \"maas-api-revoke-sub-2e1f5b5f-hg58l\" (UID: \"2d35530f-1a3b-485d-a41f-dd20fbcb5b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hg58l"
Oct 30 21:03:33.930426 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:33.930383    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d35530f-1a3b-485d-a41f-dd20fbcb5b01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-hg58l\" (UID: \"2d35530f-1a3b-485d-a41f-dd20fbcb5b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hg58l"
Oct 30 21:03:34.031751 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:34.031719    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq2tr\" (UniqueName: \"kubernetes.io/projected/2d35530f-1a3b-485d-a41f-dd20fbcb5b01-kube-api-access-pq2tr\") pod \"maas-api-revoke-sub-2e1f5b5f-hg58l\" (UID: \"2d35530f-1a3b-485d-a41f-dd20fbcb5b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hg58l"
Oct 30 21:03:34.031936 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:34.031773    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d35530f-1a3b-485d-a41f-dd20fbcb5b01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-hg58l\" (UID: \"2d35530f-1a3b-485d-a41f-dd20fbcb5b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hg58l"
Oct 30 21:03:34.031936 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:34.031879    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d35530f-1a3b-485d-a41f-dd20fbcb5b01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-hg58l\" (UID: \"2d35530f-1a3b-485d-a41f-dd20fbcb5b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hg58l"
Oct 30 21:03:34.032510 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:34.032492    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d35530f-1a3b-485d-a41f-dd20fbcb5b01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-hg58l\" (UID: \"2d35530f-1a3b-485d-a41f-dd20fbcb5b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hg58l"
Oct 30 21:03:34.034228 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:34.034209    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d35530f-1a3b-485d-a41f-dd20fbcb5b01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-hg58l\" (UID: \"2d35530f-1a3b-485d-a41f-dd20fbcb5b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hg58l"
Oct 30 21:03:34.090410 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:34.090341    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq2tr\" (UniqueName: \"kubernetes.io/projected/2d35530f-1a3b-485d-a41f-dd20fbcb5b01-kube-api-access-pq2tr\") pod \"maas-api-revoke-sub-2e1f5b5f-hg58l\" (UID: \"2d35530f-1a3b-485d-a41f-dd20fbcb5b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hg58l"
Oct 30 21:03:34.131629 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:34.131598    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hg58l"
Oct 30 21:03:34.508986 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:34.508955    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hg58l"]
Oct 30 21:03:34.511090 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:03:34.511054    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d35530f_1a3b_485d_a41f_dd20fbcb5b01.slice/crio-823f5088e5bdd32c92fd1a86316352fd8ae984fade9a946a54906595f0a98b39 WatchSource:0}: Error finding container 823f5088e5bdd32c92fd1a86316352fd8ae984fade9a946a54906595f0a98b39: Status 404 returned error can't find the container with id 823f5088e5bdd32c92fd1a86316352fd8ae984fade9a946a54906595f0a98b39
Oct 30 21:03:35.289487 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:35.289452    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:35.347670 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:35.347627    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hg58l" event={"ID":"2d35530f-1a3b-485d-a41f-dd20fbcb5b01","Type":"ContainerStarted","Data":"a05103928f84aabe72fb6f57ecb8b2b1aa3148eace7742bffd02449050a258e2"}
Oct 30 21:03:35.347867 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:35.347731    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hg58l" event={"ID":"2d35530f-1a3b-485d-a41f-dd20fbcb5b01","Type":"ContainerStarted","Data":"823f5088e5bdd32c92fd1a86316352fd8ae984fade9a946a54906595f0a98b39"}
Oct 30 21:03:35.462331 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:35.462281    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hg58l" podStartSLOduration=2.462262933 podStartE2EDuration="2.462262933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:03:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:03:35.461938687 +0000 UTC m=+2227.378960224" watchObservedRunningTime="2026-10-30 21:03:35.462262933 +0000 UTC m=+2227.379284478"
Oct 30 21:03:36.354752 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:36.354716    5071 generic.go:356] "Generic (PLEG): container finished" podID="2d35530f-1a3b-485d-a41f-dd20fbcb5b01" containerID="a05103928f84aabe72fb6f57ecb8b2b1aa3148eace7742bffd02449050a258e2" exitCode=0
Oct 30 21:03:36.355193 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:36.354799    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hg58l" event={"ID":"2d35530f-1a3b-485d-a41f-dd20fbcb5b01","Type":"ContainerDied","Data":"a05103928f84aabe72fb6f57ecb8b2b1aa3148eace7742bffd02449050a258e2"}
Oct 30 21:03:37.518706 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:37.518680    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hg58l"
Oct 30 21:03:37.569005 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:37.568967    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d35530f-1a3b-485d-a41f-dd20fbcb5b01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d35530f-1a3b-485d-a41f-dd20fbcb5b01-maas-api-xks-ca\") pod \"2d35530f-1a3b-485d-a41f-dd20fbcb5b01\" (UID: \"2d35530f-1a3b-485d-a41f-dd20fbcb5b01\") "
Oct 30 21:03:37.569176 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:37.569080    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d35530f-1a3b-485d-a41f-dd20fbcb5b01-kube-api-access-pq2tr\" (UniqueName: \"kubernetes.io/projected/2d35530f-1a3b-485d-a41f-dd20fbcb5b01-kube-api-access-pq2tr\") pod \"2d35530f-1a3b-485d-a41f-dd20fbcb5b01\" (UID: \"2d35530f-1a3b-485d-a41f-dd20fbcb5b01\") "
Oct 30 21:03:37.569176 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:37.569133    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d35530f-1a3b-485d-a41f-dd20fbcb5b01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d35530f-1a3b-485d-a41f-dd20fbcb5b01-maas-api-openshift-ca\") pod \"2d35530f-1a3b-485d-a41f-dd20fbcb5b01\" (UID: \"2d35530f-1a3b-485d-a41f-dd20fbcb5b01\") "
Oct 30 21:03:37.569522 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:37.569499    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d35530f-1a3b-485d-a41f-dd20fbcb5b01-maas-api-openshift-ca" pod "2d35530f-1a3b-485d-a41f-dd20fbcb5b01" (UID: "2d35530f-1a3b-485d-a41f-dd20fbcb5b01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:37.571309 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:37.571277    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d35530f-1a3b-485d-a41f-dd20fbcb5b01-kube-api-access-pq2tr" pod "2d35530f-1a3b-485d-a41f-dd20fbcb5b01" (UID: "2d35530f-1a3b-485d-a41f-dd20fbcb5b01"). InnerVolumeSpecName "kube-api-access-pq2tr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:37.571455 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:37.571317    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d35530f-1a3b-485d-a41f-dd20fbcb5b01-maas-api-xks-ca" pod "2d35530f-1a3b-485d-a41f-dd20fbcb5b01" (UID: "2d35530f-1a3b-485d-a41f-dd20fbcb5b01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:03:37.670574 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:37.670491    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq2tr\" (UniqueName: \"kubernetes.io/projected/2d35530f-1a3b-485d-a41f-dd20fbcb5b01-kube-api-access-pq2tr\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:37.670574 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:37.670534    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d35530f-1a3b-485d-a41f-dd20fbcb5b01-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:37.670574 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:37.670555    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d35530f-1a3b-485d-a41f-dd20fbcb5b01-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:37.791500 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:37.791466    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:38.366867 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:38.366837    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hg58l"
Oct 30 21:03:38.366867 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:38.366849    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hg58l" event={"ID":"2d35530f-1a3b-485d-a41f-dd20fbcb5b01","Type":"ContainerDied","Data":"823f5088e5bdd32c92fd1a86316352fd8ae984fade9a946a54906595f0a98b39"}
Oct 30 21:03:38.367077 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:38.366899    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="823f5088e5bdd32c92fd1a86316352fd8ae984fade9a946a54906595f0a98b39"
Oct 30 21:03:43.214164 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:43.214128    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:03:46.359063 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:46.359027    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clfv7"]
Oct 30 21:03:46.359642 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:46.359626    5071 cpu_manager.go:414] "containerMap: removing container" podUID="2d35530f-1a3b-485d-a41f-dd20fbcb5b01" containerName="revoke-keys"
Oct 30 21:03:46.359688 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:46.359645    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d35530f-1a3b-485d-a41f-dd20fbcb5b01" containerName="revoke-keys"
Oct 30 21:03:46.359784 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:46.359772    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d35530f-1a3b-485d-a41f-dd20fbcb5b01" containerName="revoke-keys"
Oct 30 21:03:46.363442 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:46.363424    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clfv7"
Oct 30 21:03:46.419943 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:46.419908    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clfv7"]
Oct 30 21:03:46.460702 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:46.460666    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87fns\" (UniqueName: \"kubernetes.io/projected/d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67-kube-api-access-87fns\") pod \"maas-api-revoke-sub-ec843018-clfv7\" (UID: \"d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clfv7"
Oct 30 21:03:46.460887 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:46.460717    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-clfv7\" (UID: \"d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clfv7"
Oct 30 21:03:46.460887 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:46.460809    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-clfv7\" (UID: \"d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clfv7"
Oct 30 21:03:46.466540 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:46.466516    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:03:46.466540 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:46.466517    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:03:46.466717 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:46.466517    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:03:46.466717 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:46.466576    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 21:03:46.562035 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:46.562003    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87fns\" (UniqueName: \"kubernetes.io/projected/d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67-kube-api-access-87fns\") pod \"maas-api-revoke-sub-ec843018-clfv7\" (UID: \"d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clfv7"
Oct 30 21:03:46.562239 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:46.562055    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-clfv7\" (UID: \"d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clfv7"
Oct 30 21:03:46.562239 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:46.562088    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-clfv7\" (UID: \"d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clfv7"
Oct 30 21:03:46.562791 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:46.562764    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-clfv7\" (UID: \"d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clfv7"
Oct 30 21:03:46.564719 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:46.564697    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-clfv7\" (UID: \"d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clfv7"
Oct 30 21:03:46.590695 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:46.590665    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:03:46.623854 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:46.623768    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87fns\" (UniqueName: \"kubernetes.io/projected/d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67-kube-api-access-87fns\") pod \"maas-api-revoke-sub-ec843018-clfv7\" (UID: \"d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clfv7"
Oct 30 21:03:46.677025 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:46.676996    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clfv7"
Oct 30 21:03:46.734227 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:03:46.734172    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:03:47.066409 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:47.066386    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clfv7"]
Oct 30 21:03:47.068238 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:03:47.068197    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c0c6eb_9df2_4c5f_84cd_d701bf7c3d67.slice/crio-a57aae155a18dd66d15fed65832ef24556e3837d29523937b43778f96fd06db8 WatchSource:0}: Error finding container a57aae155a18dd66d15fed65832ef24556e3837d29523937b43778f96fd06db8: Status 404 returned error can't find the container with id a57aae155a18dd66d15fed65832ef24556e3837d29523937b43778f96fd06db8
Oct 30 21:03:47.415557 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:47.415523    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clfv7" event={"ID":"d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67","Type":"ContainerStarted","Data":"876a9d31aa718903a53e7c4d72d867e612daf5c65efeb38ab0e4d07ead9d5a8e"}
Oct 30 21:03:47.415970 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:47.415585    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clfv7" event={"ID":"d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67","Type":"ContainerStarted","Data":"a57aae155a18dd66d15fed65832ef24556e3837d29523937b43778f96fd06db8"}
Oct 30 21:03:47.531604 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:47.531555    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clfv7" podStartSLOduration=1.531541132 podStartE2EDuration="1.531541132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:03:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:03:47.531287438 +0000 UTC m=+2239.448308974" watchObservedRunningTime="2026-10-30 21:03:47.531541132 +0000 UTC m=+2239.448562667"
Oct 30 21:03:48.427777 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:48.427739    5071 generic.go:356] "Generic (PLEG): container finished" podID="d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67" containerID="876a9d31aa718903a53e7c4d72d867e612daf5c65efeb38ab0e4d07ead9d5a8e" exitCode=0
Oct 30 21:03:48.428226 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:48.427813    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clfv7" event={"ID":"d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67","Type":"ContainerDied","Data":"876a9d31aa718903a53e7c4d72d867e612daf5c65efeb38ab0e4d07ead9d5a8e"}
Oct 30 21:03:49.594567 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:49.594541    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clfv7"
Oct 30 21:03:49.693983 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:49.693894    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67-maas-api-openshift-ca\") pod \"d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67\" (UID: \"d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67\") "
Oct 30 21:03:49.693983 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:49.693968    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67-kube-api-access-87fns\" (UniqueName: \"kubernetes.io/projected/d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67-kube-api-access-87fns\") pod \"d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67\" (UID: \"d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67\") "
Oct 30 21:03:49.694233 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:49.694069    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67-maas-api-xks-ca\") pod \"d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67\" (UID: \"d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67\") "
Oct 30 21:03:49.694406 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:49.694378    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67-maas-api-openshift-ca" pod "d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67" (UID: "d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:49.696434 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:49.696406    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67-kube-api-access-87fns" pod "d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67" (UID: "d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67"). InnerVolumeSpecName "kube-api-access-87fns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:49.696677 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:49.696646    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67-maas-api-xks-ca" pod "d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67" (UID: "d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:03:49.795123 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:49.795085    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:49.795123 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:49.795119    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87fns\" (UniqueName: \"kubernetes.io/projected/d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67-kube-api-access-87fns\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:49.795397 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:49.795134    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:50.444524 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:50.444495    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clfv7"
Oct 30 21:03:50.444524 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:50.444508    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clfv7" event={"ID":"d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67","Type":"ContainerDied","Data":"a57aae155a18dd66d15fed65832ef24556e3837d29523937b43778f96fd06db8"}
Oct 30 21:03:50.444761 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:03:50.444558    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a57aae155a18dd66d15fed65832ef24556e3837d29523937b43778f96fd06db8"
Oct 30 21:04:02.196413 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:02.196376    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:03.684149 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:03.684106    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:04:06.645858 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:06.644923    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-xhg2f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:04:06.645858 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:06.644993    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f" podUID="562c4391-03f9-470d-a007-b4931bd3f421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:04:06.690857 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:06.690167    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-pmwpk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:04:06.690857 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:06.690254    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk" podUID="a367ee40-c5ea-44f3-92c6-d1b5cf1141b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:04:16.729222 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:04:16.729181    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:04:21.935500 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:21.935464    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-flpwp"]
Oct 30 21:04:21.935923 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:21.935901    5071 cpu_manager.go:414] "containerMap: removing container" podUID="d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67" containerName="revoke-keys"
Oct 30 21:04:21.935923 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:21.935914    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67" containerName="revoke-keys"
Oct 30 21:04:21.936023 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:21.936013    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67" containerName="revoke-keys"
Oct 30 21:04:21.939012 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:21.938997    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-flpwp"
Oct 30 21:04:21.992380 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:21.992351    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:04:22.041325 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:22.041297    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 21:04:22.041587 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:22.041565    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:04:22.041677 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:22.041616    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:04:22.043971 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:22.043954    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-flpwp"]
Oct 30 21:04:22.055393 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:22.055370    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-flpwp\" (UID: \"e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-flpwp"
Oct 30 21:04:22.055512 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:22.055415    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-flpwp\" (UID: \"e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-flpwp"
Oct 30 21:04:22.055566 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:22.055550    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlhsx\" (UniqueName: \"kubernetes.io/projected/e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621-kube-api-access-vlhsx\") pod \"maas-api-revoke-sub-046e1343-flpwp\" (UID: \"e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-flpwp"
Oct 30 21:04:22.156483 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:22.156448    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlhsx\" (UniqueName: \"kubernetes.io/projected/e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621-kube-api-access-vlhsx\") pod \"maas-api-revoke-sub-046e1343-flpwp\" (UID: \"e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-flpwp"
Oct 30 21:04:22.156666 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:22.156498    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-flpwp\" (UID: \"e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-flpwp"
Oct 30 21:04:22.156666 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:22.156541    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-flpwp\" (UID: \"e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-flpwp"
Oct 30 21:04:22.157310 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:22.157284    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-flpwp\" (UID: \"e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-flpwp"
Oct 30 21:04:22.159334 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:22.159308    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-flpwp\" (UID: \"e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-flpwp"
Oct 30 21:04:22.215995 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:22.215940    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlhsx\" (UniqueName: \"kubernetes.io/projected/e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621-kube-api-access-vlhsx\") pod \"maas-api-revoke-sub-046e1343-flpwp\" (UID: \"e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-flpwp"
Oct 30 21:04:22.251734 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:22.251702    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-flpwp"
Oct 30 21:04:22.623920 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:22.623894    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-flpwp"]
Oct 30 21:04:22.625606 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:04:22.625573    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6a4cf9c_bd67_4cc8_bbdf_e648a0dc7621.slice/crio-5acc4f7bda946bf411d178bc312cea375d840bb76844cede366b96c5536430d1 WatchSource:0}: Error finding container 5acc4f7bda946bf411d178bc312cea375d840bb76844cede366b96c5536430d1: Status 404 returned error can't find the container with id 5acc4f7bda946bf411d178bc312cea375d840bb76844cede366b96c5536430d1
Oct 30 21:04:22.644417 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:22.644385    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-flpwp" event={"ID":"e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621","Type":"ContainerStarted","Data":"5acc4f7bda946bf411d178bc312cea375d840bb76844cede366b96c5536430d1"}
Oct 30 21:04:23.651732 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:23.651696    5071 generic.go:356] "Generic (PLEG): container finished" podID="e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621" containerID="d6df5ba86e128843c6993d2c39afbcbd55de74bbcd8db6a15c8e3c7724d12ec4" exitCode=0
Oct 30 21:04:23.652173 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:23.651779    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-flpwp" event={"ID":"e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621","Type":"ContainerDied","Data":"d6df5ba86e128843c6993d2c39afbcbd55de74bbcd8db6a15c8e3c7724d12ec4"}
Oct 30 21:04:24.811329 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:24.811304    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-flpwp"
Oct 30 21:04:24.881740 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:24.881713    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621-kube-api-access-vlhsx\" (UniqueName: \"kubernetes.io/projected/e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621-kube-api-access-vlhsx\") pod \"e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621\" (UID: \"e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621\") "
Oct 30 21:04:24.881925 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:24.881869    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621-maas-api-openshift-ca\") pod \"e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621\" (UID: \"e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621\") "
Oct 30 21:04:24.881993 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:24.881923    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621-maas-api-xks-ca\") pod \"e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621\" (UID: \"e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621\") "
Oct 30 21:04:24.882266 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:24.882239    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621-maas-api-openshift-ca" pod "e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621" (UID: "e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:04:24.884029 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:24.884006    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621-kube-api-access-vlhsx" pod "e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621" (UID: "e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621"). InnerVolumeSpecName "kube-api-access-vlhsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:04:24.884146 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:24.884050    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621-maas-api-xks-ca" pod "e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621" (UID: "e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:04:24.983367 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:24.983276    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:24.983367 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:24.983308    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlhsx\" (UniqueName: \"kubernetes.io/projected/e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621-kube-api-access-vlhsx\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:24.983367 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:24.983323    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:25.664304 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:25.664270    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-flpwp"
Oct 30 21:04:25.664489 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:25.664269    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-flpwp" event={"ID":"e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621","Type":"ContainerDied","Data":"5acc4f7bda946bf411d178bc312cea375d840bb76844cede366b96c5536430d1"}
Oct 30 21:04:25.664489 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:25.664390    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5acc4f7bda946bf411d178bc312cea375d840bb76844cede366b96c5536430d1"
Oct 30 21:04:32.697453 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:32.697420    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kz6v7"]
Oct 30 21:04:32.698091 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:32.698070    5071 cpu_manager.go:414] "containerMap: removing container" podUID="e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621" containerName="revoke-keys"
Oct 30 21:04:32.698183 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:32.698094    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621" containerName="revoke-keys"
Oct 30 21:04:32.698263 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:32.698250    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621" containerName="revoke-keys"
Oct 30 21:04:32.701871 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:32.701850    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kz6v7"
Oct 30 21:04:32.758194 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:32.758157    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kz6v7"]
Oct 30 21:04:32.804416 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:32.804376    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:04:32.804579 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:32.804410    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:04:32.804579 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:32.804439    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:04:32.804579 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:32.804453    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 21:04:32.855600 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:32.855559    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84pbq\" (UniqueName: \"kubernetes.io/projected/01ae32d3-9b0b-49ab-a1bc-3146f79084d7-kube-api-access-84pbq\") pod \"maas-api-revoke-sub-d3fba056-kz6v7\" (UID: \"01ae32d3-9b0b-49ab-a1bc-3146f79084d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kz6v7"
Oct 30 21:04:32.855790 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:32.855631    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01ae32d3-9b0b-49ab-a1bc-3146f79084d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-kz6v7\" (UID: \"01ae32d3-9b0b-49ab-a1bc-3146f79084d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kz6v7"
Oct 30 21:04:32.855790 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:32.855710    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01ae32d3-9b0b-49ab-a1bc-3146f79084d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-kz6v7\" (UID: \"01ae32d3-9b0b-49ab-a1bc-3146f79084d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kz6v7"
Oct 30 21:04:32.956464 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:32.956373    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01ae32d3-9b0b-49ab-a1bc-3146f79084d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-kz6v7\" (UID: \"01ae32d3-9b0b-49ab-a1bc-3146f79084d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kz6v7"
Oct 30 21:04:32.956464 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:32.956444    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01ae32d3-9b0b-49ab-a1bc-3146f79084d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-kz6v7\" (UID: \"01ae32d3-9b0b-49ab-a1bc-3146f79084d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kz6v7"
Oct 30 21:04:32.956710 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:32.956549    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84pbq\" (UniqueName: \"kubernetes.io/projected/01ae32d3-9b0b-49ab-a1bc-3146f79084d7-kube-api-access-84pbq\") pod \"maas-api-revoke-sub-d3fba056-kz6v7\" (UID: \"01ae32d3-9b0b-49ab-a1bc-3146f79084d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kz6v7"
Oct 30 21:04:32.957145 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:32.957112    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01ae32d3-9b0b-49ab-a1bc-3146f79084d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-kz6v7\" (UID: \"01ae32d3-9b0b-49ab-a1bc-3146f79084d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kz6v7"
Oct 30 21:04:32.958949 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:32.958931    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01ae32d3-9b0b-49ab-a1bc-3146f79084d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-kz6v7\" (UID: \"01ae32d3-9b0b-49ab-a1bc-3146f79084d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kz6v7"
Oct 30 21:04:33.014515 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:33.014490    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84pbq\" (UniqueName: \"kubernetes.io/projected/01ae32d3-9b0b-49ab-a1bc-3146f79084d7-kube-api-access-84pbq\") pod \"maas-api-revoke-sub-d3fba056-kz6v7\" (UID: \"01ae32d3-9b0b-49ab-a1bc-3146f79084d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kz6v7"
Oct 30 21:04:33.016306 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:33.016290    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kz6v7"
Oct 30 21:04:33.393202 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:33.393177    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kz6v7"]
Oct 30 21:04:33.394737 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:04:33.394708    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ae32d3_9b0b_49ab_a1bc_3146f79084d7.slice/crio-165035d2bb52d8d60225df10b84343f22e57dba087b1a242c093aeb5465e690e WatchSource:0}: Error finding container 165035d2bb52d8d60225df10b84343f22e57dba087b1a242c093aeb5465e690e: Status 404 returned error can't find the container with id 165035d2bb52d8d60225df10b84343f22e57dba087b1a242c093aeb5465e690e
Oct 30 21:04:33.715788 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:33.715746    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kz6v7" event={"ID":"01ae32d3-9b0b-49ab-a1bc-3146f79084d7","Type":"ContainerStarted","Data":"165035d2bb52d8d60225df10b84343f22e57dba087b1a242c093aeb5465e690e"}
Oct 30 21:04:34.722934 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:34.722812    5071 generic.go:356] "Generic (PLEG): container finished" podID="01ae32d3-9b0b-49ab-a1bc-3146f79084d7" containerID="0f3fa36b0d55e648ef0fb9d4d8ec347da7f10fc3b835233cc7698c2b61ee3bb1" exitCode=0
Oct 30 21:04:34.722934 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:34.722883    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kz6v7" event={"ID":"01ae32d3-9b0b-49ab-a1bc-3146f79084d7","Type":"ContainerDied","Data":"0f3fa36b0d55e648ef0fb9d4d8ec347da7f10fc3b835233cc7698c2b61ee3bb1"}
Oct 30 21:04:34.971530 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:34.971494    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:04:35.646234 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:35.646198    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:35.954329 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:35.954294    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kz6v7"
Oct 30 21:04:36.086262 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:36.086221    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01ae32d3-9b0b-49ab-a1bc-3146f79084d7-kube-api-access-84pbq\" (UniqueName: \"kubernetes.io/projected/01ae32d3-9b0b-49ab-a1bc-3146f79084d7-kube-api-access-84pbq\") pod \"01ae32d3-9b0b-49ab-a1bc-3146f79084d7\" (UID: \"01ae32d3-9b0b-49ab-a1bc-3146f79084d7\") "
Oct 30 21:04:36.086470 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:36.086347    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01ae32d3-9b0b-49ab-a1bc-3146f79084d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01ae32d3-9b0b-49ab-a1bc-3146f79084d7-maas-api-openshift-ca\") pod \"01ae32d3-9b0b-49ab-a1bc-3146f79084d7\" (UID: \"01ae32d3-9b0b-49ab-a1bc-3146f79084d7\") "
Oct 30 21:04:36.086470 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:36.086392    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01ae32d3-9b0b-49ab-a1bc-3146f79084d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01ae32d3-9b0b-49ab-a1bc-3146f79084d7-maas-api-xks-ca\") pod \"01ae32d3-9b0b-49ab-a1bc-3146f79084d7\" (UID: \"01ae32d3-9b0b-49ab-a1bc-3146f79084d7\") "
Oct 30 21:04:36.087905 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:36.087872    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ae32d3-9b0b-49ab-a1bc-3146f79084d7-maas-api-openshift-ca" pod "01ae32d3-9b0b-49ab-a1bc-3146f79084d7" (UID: "01ae32d3-9b0b-49ab-a1bc-3146f79084d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:04:36.089871 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:36.089806    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ae32d3-9b0b-49ab-a1bc-3146f79084d7-maas-api-xks-ca" pod "01ae32d3-9b0b-49ab-a1bc-3146f79084d7" (UID: "01ae32d3-9b0b-49ab-a1bc-3146f79084d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:04:36.097704 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:36.097676    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ae32d3-9b0b-49ab-a1bc-3146f79084d7-kube-api-access-84pbq" pod "01ae32d3-9b0b-49ab-a1bc-3146f79084d7" (UID: "01ae32d3-9b0b-49ab-a1bc-3146f79084d7"). InnerVolumeSpecName "kube-api-access-84pbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:04:36.188190 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:36.188103    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84pbq\" (UniqueName: \"kubernetes.io/projected/01ae32d3-9b0b-49ab-a1bc-3146f79084d7-kube-api-access-84pbq\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:36.188190 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:36.188140    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01ae32d3-9b0b-49ab-a1bc-3146f79084d7-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:36.188190 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:36.188157    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01ae32d3-9b0b-49ab-a1bc-3146f79084d7-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:36.773539 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:36.773499    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kz6v7" event={"ID":"01ae32d3-9b0b-49ab-a1bc-3146f79084d7","Type":"ContainerDied","Data":"165035d2bb52d8d60225df10b84343f22e57dba087b1a242c093aeb5465e690e"}
Oct 30 21:04:36.773863 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:36.773841    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="165035d2bb52d8d60225df10b84343f22e57dba087b1a242c093aeb5465e690e"
Oct 30 21:04:36.774074 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:36.774060    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kz6v7"
Oct 30 21:04:39.336774 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:39.336698    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:43.550717 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:43.550679    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:04:45.566683 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:45.566647    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:04:46.728022 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:04:46.727985    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:04:58.477747 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:04:58.477705    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:05:10.634049 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:10.634018    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shbg5"]
Oct 30 21:05:10.634683 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:10.634662    5071 cpu_manager.go:414] "containerMap: removing container" podUID="01ae32d3-9b0b-49ab-a1bc-3146f79084d7" containerName="revoke-keys"
Oct 30 21:05:10.634786 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:10.634684    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01ae32d3-9b0b-49ab-a1bc-3146f79084d7" containerName="revoke-keys"
Oct 30 21:05:10.634878 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:10.634868    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="01ae32d3-9b0b-49ab-a1bc-3146f79084d7" containerName="revoke-keys"
Oct 30 21:05:10.639362 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:10.639342    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shbg5"
Oct 30 21:05:10.695481 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:10.695439    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shbg5"]
Oct 30 21:05:10.711349 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:10.711317    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnqr5\" (UniqueName: \"kubernetes.io/projected/c84605e5-be76-47c2-8fe3-e439135d3396-kube-api-access-xnqr5\") pod \"maas-api-revoke-sub-ae594979-shbg5\" (UID: \"c84605e5-be76-47c2-8fe3-e439135d3396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shbg5"
Oct 30 21:05:10.711518 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:10.711452    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84605e5-be76-47c2-8fe3-e439135d3396-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-shbg5\" (UID: \"c84605e5-be76-47c2-8fe3-e439135d3396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shbg5"
Oct 30 21:05:10.711594 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:10.711512    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84605e5-be76-47c2-8fe3-e439135d3396-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-shbg5\" (UID: \"c84605e5-be76-47c2-8fe3-e439135d3396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shbg5"
Oct 30 21:05:10.741618 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:10.741589    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:05:10.741618 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:10.741609    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:05:10.741799 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:10.741600    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:05:10.742771 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:10.742755    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 21:05:10.812306 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:10.812267    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnqr5\" (UniqueName: \"kubernetes.io/projected/c84605e5-be76-47c2-8fe3-e439135d3396-kube-api-access-xnqr5\") pod \"maas-api-revoke-sub-ae594979-shbg5\" (UID: \"c84605e5-be76-47c2-8fe3-e439135d3396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shbg5"
Oct 30 21:05:10.812492 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:10.812336    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84605e5-be76-47c2-8fe3-e439135d3396-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-shbg5\" (UID: \"c84605e5-be76-47c2-8fe3-e439135d3396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shbg5"
Oct 30 21:05:10.812492 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:10.812372    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84605e5-be76-47c2-8fe3-e439135d3396-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-shbg5\" (UID: \"c84605e5-be76-47c2-8fe3-e439135d3396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shbg5"
Oct 30 21:05:10.812998 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:10.812971    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84605e5-be76-47c2-8fe3-e439135d3396-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-shbg5\" (UID: \"c84605e5-be76-47c2-8fe3-e439135d3396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shbg5"
Oct 30 21:05:10.815113 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:10.815082    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84605e5-be76-47c2-8fe3-e439135d3396-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-shbg5\" (UID: \"c84605e5-be76-47c2-8fe3-e439135d3396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shbg5"
Oct 30 21:05:10.871216 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:10.871179    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnqr5\" (UniqueName: \"kubernetes.io/projected/c84605e5-be76-47c2-8fe3-e439135d3396-kube-api-access-xnqr5\") pod \"maas-api-revoke-sub-ae594979-shbg5\" (UID: \"c84605e5-be76-47c2-8fe3-e439135d3396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shbg5"
Oct 30 21:05:10.956232 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:10.956158    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shbg5"
Oct 30 21:05:11.330906 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:11.330884    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shbg5"]
Oct 30 21:05:11.332872 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:05:11.332845    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc84605e5_be76_47c2_8fe3_e439135d3396.slice/crio-b5681ca6ad0087b375ae8c41d8dcc236609c48a1c49581430d6ae962ca3b0d85 WatchSource:0}: Error finding container b5681ca6ad0087b375ae8c41d8dcc236609c48a1c49581430d6ae962ca3b0d85: Status 404 returned error can't find the container with id b5681ca6ad0087b375ae8c41d8dcc236609c48a1c49581430d6ae962ca3b0d85
Oct 30 21:05:12.027352 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:12.027258    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shbg5" event={"ID":"c84605e5-be76-47c2-8fe3-e439135d3396","Type":"ContainerStarted","Data":"b64e75088afaa915b7804b10ab2488a7ed6080182e8754aea538c46f48885634"}
Oct 30 21:05:12.027352 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:12.027310    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shbg5" event={"ID":"c84605e5-be76-47c2-8fe3-e439135d3396","Type":"ContainerStarted","Data":"b5681ca6ad0087b375ae8c41d8dcc236609c48a1c49581430d6ae962ca3b0d85"}
Oct 30 21:05:12.051893 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:12.051862    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7fb8880-jxtnj"]
Oct 30 21:05:12.101341 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:12.101313    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7fb8880-jxtnj"]
Oct 30 21:05:12.161962 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:12.161902    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shbg5" podStartSLOduration=2.161888999 podStartE2EDuration="2.161888999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:05:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:05:12.160440004 +0000 UTC m=+2324.077461553" watchObservedRunningTime="2026-10-30 21:05:12.161888999 +0000 UTC m=+2324.078910535"
Oct 30 21:05:12.201265 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:12.201234    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dc29059-37fb-4f25-8809-148d6115a8a5" path="/var/lib/kubelet/pods/6dc29059-37fb-4f25-8809-148d6115a8a5/volumes"
Oct 30 21:05:13.034384 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:13.034347    5071 generic.go:356] "Generic (PLEG): container finished" podID="c84605e5-be76-47c2-8fe3-e439135d3396" containerID="b64e75088afaa915b7804b10ab2488a7ed6080182e8754aea538c46f48885634" exitCode=0
Oct 30 21:05:13.034846 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:13.034439    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shbg5" event={"ID":"c84605e5-be76-47c2-8fe3-e439135d3396","Type":"ContainerDied","Data":"b64e75088afaa915b7804b10ab2488a7ed6080182e8754aea538c46f48885634"}
Oct 30 21:05:14.198483 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:14.198459    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shbg5"
Oct 30 21:05:14.246058 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:14.246019    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c84605e5-be76-47c2-8fe3-e439135d3396-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84605e5-be76-47c2-8fe3-e439135d3396-maas-api-openshift-ca\") pod \"c84605e5-be76-47c2-8fe3-e439135d3396\" (UID: \"c84605e5-be76-47c2-8fe3-e439135d3396\") "
Oct 30 21:05:14.246215 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:14.246103    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c84605e5-be76-47c2-8fe3-e439135d3396-kube-api-access-xnqr5\" (UniqueName: \"kubernetes.io/projected/c84605e5-be76-47c2-8fe3-e439135d3396-kube-api-access-xnqr5\") pod \"c84605e5-be76-47c2-8fe3-e439135d3396\" (UID: \"c84605e5-be76-47c2-8fe3-e439135d3396\") "
Oct 30 21:05:14.246215 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:14.246165    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c84605e5-be76-47c2-8fe3-e439135d3396-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84605e5-be76-47c2-8fe3-e439135d3396-maas-api-xks-ca\") pod \"c84605e5-be76-47c2-8fe3-e439135d3396\" (UID: \"c84605e5-be76-47c2-8fe3-e439135d3396\") "
Oct 30 21:05:14.246420 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:14.246396    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c84605e5-be76-47c2-8fe3-e439135d3396-maas-api-openshift-ca" pod "c84605e5-be76-47c2-8fe3-e439135d3396" (UID: "c84605e5-be76-47c2-8fe3-e439135d3396"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:05:14.246533 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:14.246517    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84605e5-be76-47c2-8fe3-e439135d3396-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:14.248310 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:14.248290    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c84605e5-be76-47c2-8fe3-e439135d3396-maas-api-xks-ca" pod "c84605e5-be76-47c2-8fe3-e439135d3396" (UID: "c84605e5-be76-47c2-8fe3-e439135d3396"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:05:14.248461 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:14.248442    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c84605e5-be76-47c2-8fe3-e439135d3396-kube-api-access-xnqr5" pod "c84605e5-be76-47c2-8fe3-e439135d3396" (UID: "c84605e5-be76-47c2-8fe3-e439135d3396"). InnerVolumeSpecName "kube-api-access-xnqr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:05:14.347139 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:14.347074    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84605e5-be76-47c2-8fe3-e439135d3396-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:14.347139 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:14.347101    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnqr5\" (UniqueName: \"kubernetes.io/projected/c84605e5-be76-47c2-8fe3-e439135d3396-kube-api-access-xnqr5\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:15.046897 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:15.046861    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shbg5" event={"ID":"c84605e5-be76-47c2-8fe3-e439135d3396","Type":"ContainerDied","Data":"b5681ca6ad0087b375ae8c41d8dcc236609c48a1c49581430d6ae962ca3b0d85"}
Oct 30 21:05:15.047079 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:15.046920    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5681ca6ad0087b375ae8c41d8dcc236609c48a1c49581430d6ae962ca3b0d85"
Oct 30 21:05:15.047079 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:15.046872    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shbg5"
Oct 30 21:05:16.727601 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:05:16.727567    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:05:23.566242 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:23.566204    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:05:33.433311 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:33.433268    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:05:35.967282 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:35.967241    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:38.815043 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:38.815006    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6pt2s"]
Oct 30 21:05:38.816245 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:38.816217    5071 cpu_manager.go:414] "containerMap: removing container" podUID="c84605e5-be76-47c2-8fe3-e439135d3396" containerName="revoke-keys"
Oct 30 21:05:38.816423 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:38.816409    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c84605e5-be76-47c2-8fe3-e439135d3396" containerName="revoke-keys"
Oct 30 21:05:38.816667 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:38.816654    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="c84605e5-be76-47c2-8fe3-e439135d3396" containerName="revoke-keys"
Oct 30 21:05:38.820621 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:38.820595    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6pt2s"
Oct 30 21:05:38.875378 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:38.875346    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6pt2s"]
Oct 30 21:05:38.921288 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:38.921253    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:05:38.921288 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:38.921279    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:05:38.921523 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:38.921308    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:05:38.923428 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:38.923412    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 21:05:38.929925 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:38.929894    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc84820e-da50-4a2d-8cb6-6624f165d9c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-6pt2s\" (UID: \"cc84820e-da50-4a2d-8cb6-6624f165d9c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6pt2s"
Oct 30 21:05:38.930055 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:38.929949    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc84820e-da50-4a2d-8cb6-6624f165d9c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-6pt2s\" (UID: \"cc84820e-da50-4a2d-8cb6-6624f165d9c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6pt2s"
Oct 30 21:05:38.930138 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:38.930119    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92llv\" (UniqueName: \"kubernetes.io/projected/cc84820e-da50-4a2d-8cb6-6624f165d9c0-kube-api-access-92llv\") pod \"maas-api-revoke-sub-a4aef7d7-6pt2s\" (UID: \"cc84820e-da50-4a2d-8cb6-6624f165d9c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6pt2s"
Oct 30 21:05:39.031224 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:39.031196    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc84820e-da50-4a2d-8cb6-6624f165d9c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-6pt2s\" (UID: \"cc84820e-da50-4a2d-8cb6-6624f165d9c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6pt2s"
Oct 30 21:05:39.031405 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:39.031289    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92llv\" (UniqueName: \"kubernetes.io/projected/cc84820e-da50-4a2d-8cb6-6624f165d9c0-kube-api-access-92llv\") pod \"maas-api-revoke-sub-a4aef7d7-6pt2s\" (UID: \"cc84820e-da50-4a2d-8cb6-6624f165d9c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6pt2s"
Oct 30 21:05:39.031405 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:39.031329    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc84820e-da50-4a2d-8cb6-6624f165d9c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-6pt2s\" (UID: \"cc84820e-da50-4a2d-8cb6-6624f165d9c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6pt2s"
Oct 30 21:05:39.032059 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:39.031987    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc84820e-da50-4a2d-8cb6-6624f165d9c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-6pt2s\" (UID: \"cc84820e-da50-4a2d-8cb6-6624f165d9c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6pt2s"
Oct 30 21:05:39.033858 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:39.033812    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc84820e-da50-4a2d-8cb6-6624f165d9c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-6pt2s\" (UID: \"cc84820e-da50-4a2d-8cb6-6624f165d9c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6pt2s"
Oct 30 21:05:39.090521 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:39.090439    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92llv\" (UniqueName: \"kubernetes.io/projected/cc84820e-da50-4a2d-8cb6-6624f165d9c0-kube-api-access-92llv\") pod \"maas-api-revoke-sub-a4aef7d7-6pt2s\" (UID: \"cc84820e-da50-4a2d-8cb6-6624f165d9c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6pt2s"
Oct 30 21:05:39.137677 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:39.137642    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6pt2s"
Oct 30 21:05:39.509618 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:39.509587    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6pt2s"]
Oct 30 21:05:39.511173 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:05:39.511141    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc84820e_da50_4a2d_8cb6_6624f165d9c0.slice/crio-a6e91c9b503f7278d858c1ca137f5d96f1eccf86e08d09618d1bb75088c1d8fb WatchSource:0}: Error finding container a6e91c9b503f7278d858c1ca137f5d96f1eccf86e08d09618d1bb75088c1d8fb: Status 404 returned error can't find the container with id a6e91c9b503f7278d858c1ca137f5d96f1eccf86e08d09618d1bb75088c1d8fb
Oct 30 21:05:40.214246 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:40.214160    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6pt2s" event={"ID":"cc84820e-da50-4a2d-8cb6-6624f165d9c0","Type":"ContainerStarted","Data":"1b54e5461f3ba1d9e7d59af536c5a8b51b4d7bbd4dca3e7159314703d784d4c7"}
Oct 30 21:05:40.214246 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:40.214221    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6pt2s" event={"ID":"cc84820e-da50-4a2d-8cb6-6624f165d9c0","Type":"ContainerStarted","Data":"a6e91c9b503f7278d858c1ca137f5d96f1eccf86e08d09618d1bb75088c1d8fb"}
Oct 30 21:05:40.330572 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:40.330512    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6pt2s" podStartSLOduration=2.330498028 podStartE2EDuration="2.330498028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:05:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:05:40.328766458 +0000 UTC m=+2352.245787995" watchObservedRunningTime="2026-10-30 21:05:40.330498028 +0000 UTC m=+2352.247519610"
Oct 30 21:05:41.221092 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:41.221060    5071 generic.go:356] "Generic (PLEG): container finished" podID="cc84820e-da50-4a2d-8cb6-6624f165d9c0" containerID="1b54e5461f3ba1d9e7d59af536c5a8b51b4d7bbd4dca3e7159314703d784d4c7" exitCode=0
Oct 30 21:05:41.221526 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:41.221119    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6pt2s" event={"ID":"cc84820e-da50-4a2d-8cb6-6624f165d9c0","Type":"ContainerDied","Data":"1b54e5461f3ba1d9e7d59af536c5a8b51b4d7bbd4dca3e7159314703d784d4c7"}
Oct 30 21:05:41.768658 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:41.768621    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:42.383409 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:42.383386    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6pt2s"
Oct 30 21:05:42.465739 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:42.465706    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc84820e-da50-4a2d-8cb6-6624f165d9c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc84820e-da50-4a2d-8cb6-6624f165d9c0-maas-api-xks-ca\") pod \"cc84820e-da50-4a2d-8cb6-6624f165d9c0\" (UID: \"cc84820e-da50-4a2d-8cb6-6624f165d9c0\") "
Oct 30 21:05:42.465929 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:42.465758    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc84820e-da50-4a2d-8cb6-6624f165d9c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc84820e-da50-4a2d-8cb6-6624f165d9c0-maas-api-openshift-ca\") pod \"cc84820e-da50-4a2d-8cb6-6624f165d9c0\" (UID: \"cc84820e-da50-4a2d-8cb6-6624f165d9c0\") "
Oct 30 21:05:42.465929 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:42.465901    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc84820e-da50-4a2d-8cb6-6624f165d9c0-kube-api-access-92llv\" (UniqueName: \"kubernetes.io/projected/cc84820e-da50-4a2d-8cb6-6624f165d9c0-kube-api-access-92llv\") pod \"cc84820e-da50-4a2d-8cb6-6624f165d9c0\" (UID: \"cc84820e-da50-4a2d-8cb6-6624f165d9c0\") "
Oct 30 21:05:42.466161 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:42.466136    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc84820e-da50-4a2d-8cb6-6624f165d9c0-maas-api-openshift-ca" pod "cc84820e-da50-4a2d-8cb6-6624f165d9c0" (UID: "cc84820e-da50-4a2d-8cb6-6624f165d9c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:05:42.468056 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:42.468037    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc84820e-da50-4a2d-8cb6-6624f165d9c0-maas-api-xks-ca" pod "cc84820e-da50-4a2d-8cb6-6624f165d9c0" (UID: "cc84820e-da50-4a2d-8cb6-6624f165d9c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:05:42.468142 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:42.468100    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc84820e-da50-4a2d-8cb6-6624f165d9c0-kube-api-access-92llv" pod "cc84820e-da50-4a2d-8cb6-6624f165d9c0" (UID: "cc84820e-da50-4a2d-8cb6-6624f165d9c0"). InnerVolumeSpecName "kube-api-access-92llv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:05:42.567481 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:42.567449    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92llv\" (UniqueName: \"kubernetes.io/projected/cc84820e-da50-4a2d-8cb6-6624f165d9c0-kube-api-access-92llv\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:42.567481 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:42.567477    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc84820e-da50-4a2d-8cb6-6624f165d9c0-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:42.567481 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:42.567487    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc84820e-da50-4a2d-8cb6-6624f165d9c0-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:43.232431 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:43.232400    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6pt2s"
Oct 30 21:05:43.232589 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:43.232452    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6pt2s" event={"ID":"cc84820e-da50-4a2d-8cb6-6624f165d9c0","Type":"ContainerDied","Data":"a6e91c9b503f7278d858c1ca137f5d96f1eccf86e08d09618d1bb75088c1d8fb"}
Oct 30 21:05:43.232589 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:43.232498    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6e91c9b503f7278d858c1ca137f5d96f1eccf86e08d09618d1bb75088c1d8fb"
Oct 30 21:05:43.882749 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:43.882709    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:45.603686 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:45.603654    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:45.816535 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:45.816501    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:05:46.726958 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:05:46.726923    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:05:51.752743 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:51.752698    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:05:59.053925 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:59.053889    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-280ee9d9-rww2g"]
Oct 30 21:05:59.106794 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:05:59.106758    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-280ee9d9-rww2g"]
Oct 30 21:06:00.218101 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:00.218065    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdef71a8-3a7d-4d75-bb49-6bb0e4da9437" path="/var/lib/kubelet/pods/cdef71a8-3a7d-4d75-bb49-6bb0e4da9437/volumes"
Oct 30 21:06:04.538776 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:04.538738    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:06:09.622212 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:09.622133    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vv5gl"]
Oct 30 21:06:09.622632 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:09.622616    5071 cpu_manager.go:414] "containerMap: removing container" podUID="cc84820e-da50-4a2d-8cb6-6624f165d9c0" containerName="revoke-keys"
Oct 30 21:06:09.622687 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:09.622634    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc84820e-da50-4a2d-8cb6-6624f165d9c0" containerName="revoke-keys"
Oct 30 21:06:09.622725 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:09.622718    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc84820e-da50-4a2d-8cb6-6624f165d9c0" containerName="revoke-keys"
Oct 30 21:06:09.626039 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:09.626024    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vv5gl"
Oct 30 21:06:09.684233 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:09.684203    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vv5gl"]
Oct 30 21:06:09.730181 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:09.730148    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:06:09.730181 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:09.730166    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:06:09.730387 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:09.730186    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:06:09.730387 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:09.730174    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 21:06:09.771999 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:09.771970    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-vv5gl\" (UID: \"b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vv5gl"
Oct 30 21:06:09.772171 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:09.772036    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq29h\" (UniqueName: \"kubernetes.io/projected/b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9-kube-api-access-gq29h\") pod \"maas-api-revoke-sub-79805ca9-vv5gl\" (UID: \"b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vv5gl"
Oct 30 21:06:09.772171 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:09.772132    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-vv5gl\" (UID: \"b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vv5gl"
Oct 30 21:06:09.873232 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:09.873145    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-vv5gl\" (UID: \"b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vv5gl"
Oct 30 21:06:09.873419 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:09.873231    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-vv5gl\" (UID: \"b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vv5gl"
Oct 30 21:06:09.873419 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:09.873280    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq29h\" (UniqueName: \"kubernetes.io/projected/b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9-kube-api-access-gq29h\") pod \"maas-api-revoke-sub-79805ca9-vv5gl\" (UID: \"b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vv5gl"
Oct 30 21:06:09.874007 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:09.873985    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-vv5gl\" (UID: \"b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vv5gl"
Oct 30 21:06:09.875693 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:09.875669    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-vv5gl\" (UID: \"b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vv5gl"
Oct 30 21:06:09.931874 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:09.931839    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq29h\" (UniqueName: \"kubernetes.io/projected/b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9-kube-api-access-gq29h\") pod \"maas-api-revoke-sub-79805ca9-vv5gl\" (UID: \"b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vv5gl"
Oct 30 21:06:09.938619 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:09.938599    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vv5gl"
Oct 30 21:06:10.317464 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:10.317439    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vv5gl"]
Oct 30 21:06:10.319157 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:06:10.319129    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb56a47dd_f9e7_4df1_8bc9_f3b296bb8be9.slice/crio-ed0c17e8b94bb6b09756e403b800e1e375afabe56b3fe5896a0ab44a87925a5c WatchSource:0}: Error finding container ed0c17e8b94bb6b09756e403b800e1e375afabe56b3fe5896a0ab44a87925a5c: Status 404 returned error can't find the container with id ed0c17e8b94bb6b09756e403b800e1e375afabe56b3fe5896a0ab44a87925a5c
Oct 30 21:06:10.431648 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:10.431614    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vv5gl" event={"ID":"b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9","Type":"ContainerStarted","Data":"ed0c17e8b94bb6b09756e403b800e1e375afabe56b3fe5896a0ab44a87925a5c"}
Oct 30 21:06:11.438881 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:11.438783    5071 generic.go:356] "Generic (PLEG): container finished" podID="b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9" containerID="eea31acbd6e4e02956845f7360b736252e02df40335b72d781752c80bf3ff7a6" exitCode=0
Oct 30 21:06:11.438881 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:11.438841    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vv5gl" event={"ID":"b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9","Type":"ContainerDied","Data":"eea31acbd6e4e02956845f7360b736252e02df40335b72d781752c80bf3ff7a6"}
Oct 30 21:06:12.595691 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:12.595666    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vv5gl"
Oct 30 21:06:12.708009 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:12.707931    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9-maas-api-openshift-ca\") pod \"b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9\" (UID: \"b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9\") "
Oct 30 21:06:12.708009 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:12.707999    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9-kube-api-access-gq29h\" (UniqueName: \"kubernetes.io/projected/b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9-kube-api-access-gq29h\") pod \"b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9\" (UID: \"b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9\") "
Oct 30 21:06:12.708241 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:12.708086    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9-maas-api-xks-ca\") pod \"b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9\" (UID: \"b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9\") "
Oct 30 21:06:12.708349 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:12.708316    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9-maas-api-openshift-ca" pod "b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9" (UID: "b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:06:12.710276 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:12.710250    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9-maas-api-xks-ca" pod "b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9" (UID: "b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:06:12.710359 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:12.710308    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9-kube-api-access-gq29h" pod "b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9" (UID: "b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9"). InnerVolumeSpecName "kube-api-access-gq29h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:12.808802 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:12.808770    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:12.808802 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:12.808799    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq29h\" (UniqueName: \"kubernetes.io/projected/b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9-kube-api-access-gq29h\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:12.808802 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:12.808809    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.451794 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:13.451755    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vv5gl" event={"ID":"b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9","Type":"ContainerDied","Data":"ed0c17e8b94bb6b09756e403b800e1e375afabe56b3fe5896a0ab44a87925a5c"}
Oct 30 21:06:13.451794 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:13.451775    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vv5gl"
Oct 30 21:06:13.452052 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:13.451811    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed0c17e8b94bb6b09756e403b800e1e375afabe56b3fe5896a0ab44a87925a5c"
Oct 30 21:06:16.727734 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:06:16.727698    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:06:24.364069 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:24.364032    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:06:27.311995 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:27.311955    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-s6hth container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:06:27.312657 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:27.312605    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s6hth" podUID="ecd40162-84fb-4838-ae3b-bafa49f9cdb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:06:28.805398 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:28.805373    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8qvb_91fe9e06-0c30-4000-bddb-2ba89cd048cc/console-operator/2.log"
Oct 30 21:06:28.813795 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:28.813758    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q4f4h_80cae79e-b60f-4efe-bb58-e77753a457e5/ovn-acl-logging/0.log"
Oct 30 21:06:28.821355 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:28.821334    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8qvb_91fe9e06-0c30-4000-bddb-2ba89cd048cc/console-operator/2.log"
Oct 30 21:06:28.828452 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:28.828433    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q4f4h_80cae79e-b60f-4efe-bb58-e77753a457e5/ovn-acl-logging/0.log"
Oct 30 21:06:33.612541 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:33.612509    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j2w5c"]
Oct 30 21:06:33.613038 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:33.613023    5071 cpu_manager.go:414] "containerMap: removing container" podUID="b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9" containerName="revoke-keys"
Oct 30 21:06:33.613095 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:33.613043    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9" containerName="revoke-keys"
Oct 30 21:06:33.613134 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:33.613111    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9" containerName="revoke-keys"
Oct 30 21:06:33.616587 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:33.616567    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j2w5c"
Oct 30 21:06:33.671192 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:33.671166    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:06:33.671524 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:33.671504    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:06:33.671665 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:33.671582    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:06:33.674481 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:33.674454    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j2w5c"]
Oct 30 21:06:33.720147 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:33.720118    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 21:06:33.757609 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:33.757572    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d70f6a39-1cde-4ac6-a351-79be7fe04a8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-j2w5c\" (UID: \"d70f6a39-1cde-4ac6-a351-79be7fe04a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j2w5c"
Oct 30 21:06:33.757793 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:33.757649    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d70f6a39-1cde-4ac6-a351-79be7fe04a8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-j2w5c\" (UID: \"d70f6a39-1cde-4ac6-a351-79be7fe04a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j2w5c"
Oct 30 21:06:33.757793 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:33.757773    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f2vn\" (UniqueName: \"kubernetes.io/projected/d70f6a39-1cde-4ac6-a351-79be7fe04a8c-kube-api-access-4f2vn\") pod \"maas-api-revoke-sub-0b7e8edb-j2w5c\" (UID: \"d70f6a39-1cde-4ac6-a351-79be7fe04a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j2w5c"
Oct 30 21:06:33.858332 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:33.858291    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f2vn\" (UniqueName: \"kubernetes.io/projected/d70f6a39-1cde-4ac6-a351-79be7fe04a8c-kube-api-access-4f2vn\") pod \"maas-api-revoke-sub-0b7e8edb-j2w5c\" (UID: \"d70f6a39-1cde-4ac6-a351-79be7fe04a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j2w5c"
Oct 30 21:06:33.858536 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:33.858353    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d70f6a39-1cde-4ac6-a351-79be7fe04a8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-j2w5c\" (UID: \"d70f6a39-1cde-4ac6-a351-79be7fe04a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j2w5c"
Oct 30 21:06:33.858536 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:33.858426    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d70f6a39-1cde-4ac6-a351-79be7fe04a8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-j2w5c\" (UID: \"d70f6a39-1cde-4ac6-a351-79be7fe04a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j2w5c"
Oct 30 21:06:33.859036 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:33.859011    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d70f6a39-1cde-4ac6-a351-79be7fe04a8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-j2w5c\" (UID: \"d70f6a39-1cde-4ac6-a351-79be7fe04a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j2w5c"
Oct 30 21:06:33.861001 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:33.860979    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d70f6a39-1cde-4ac6-a351-79be7fe04a8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-j2w5c\" (UID: \"d70f6a39-1cde-4ac6-a351-79be7fe04a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j2w5c"
Oct 30 21:06:33.916552 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:33.916487    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f2vn\" (UniqueName: \"kubernetes.io/projected/d70f6a39-1cde-4ac6-a351-79be7fe04a8c-kube-api-access-4f2vn\") pod \"maas-api-revoke-sub-0b7e8edb-j2w5c\" (UID: \"d70f6a39-1cde-4ac6-a351-79be7fe04a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j2w5c"
Oct 30 21:06:33.930333 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:33.930310    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j2w5c"
Oct 30 21:06:33.992127 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:33.988309    5071 scope.go:122] "RemoveContainer" containerID="f9c32a58e76435c65e228f3e14d6e9c3f2fc233ef4bcc88168054f00c5a89280"
Oct 30 21:06:34.004865 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:34.004744    5071 scope.go:122] "RemoveContainer" containerID="6f0b271a462d319d544e42223abbacf9455cca7a90e421dd824cd09d85501dd6"
Oct 30 21:06:34.014165 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:34.014139    5071 scope.go:122] "RemoveContainer" containerID="b2128feb84214f88ac6416425ea62b831fb75ab5e78e01d8230e9282b163297a"
Oct 30 21:06:34.302850 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:34.302802    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j2w5c"]
Oct 30 21:06:34.304141 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:06:34.304114    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd70f6a39_1cde_4ac6_a351_79be7fe04a8c.slice/crio-d5a9a0379a9075eab004d5f5fafce4d58d9488d5440bda0f5282cfdd8f994d86 WatchSource:0}: Error finding container d5a9a0379a9075eab004d5f5fafce4d58d9488d5440bda0f5282cfdd8f994d86: Status 404 returned error can't find the container with id d5a9a0379a9075eab004d5f5fafce4d58d9488d5440bda0f5282cfdd8f994d86
Oct 30 21:06:34.657995 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:34.657965    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j2w5c" event={"ID":"d70f6a39-1cde-4ac6-a351-79be7fe04a8c","Type":"ContainerStarted","Data":"50e27ad7160c3ca4b4e990f2e2c7b2d34342c048ac53c4699df9a6a6af019e63"}
Oct 30 21:06:34.658329 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:34.658014    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j2w5c" event={"ID":"d70f6a39-1cde-4ac6-a351-79be7fe04a8c","Type":"ContainerStarted","Data":"d5a9a0379a9075eab004d5f5fafce4d58d9488d5440bda0f5282cfdd8f994d86"}
Oct 30 21:06:34.773988 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:34.773942    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j2w5c" podStartSLOduration=1.77392759 podStartE2EDuration="1.77392759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:34.772648614 +0000 UTC m=+2406.689670149" watchObservedRunningTime="2026-10-30 21:06:34.77392759 +0000 UTC m=+2406.690949125"
Oct 30 21:06:35.665097 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:35.665060    5071 generic.go:356] "Generic (PLEG): container finished" podID="d70f6a39-1cde-4ac6-a351-79be7fe04a8c" containerID="50e27ad7160c3ca4b4e990f2e2c7b2d34342c048ac53c4699df9a6a6af019e63" exitCode=0
Oct 30 21:06:35.665536 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:35.665121    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j2w5c" event={"ID":"d70f6a39-1cde-4ac6-a351-79be7fe04a8c","Type":"ContainerDied","Data":"50e27ad7160c3ca4b4e990f2e2c7b2d34342c048ac53c4699df9a6a6af019e63"}
Oct 30 21:06:35.765428 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:35.765391    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:06:36.814393 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:36.814371    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j2w5c"
Oct 30 21:06:36.884598 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:36.884574    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d70f6a39-1cde-4ac6-a351-79be7fe04a8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d70f6a39-1cde-4ac6-a351-79be7fe04a8c-maas-api-openshift-ca\") pod \"d70f6a39-1cde-4ac6-a351-79be7fe04a8c\" (UID: \"d70f6a39-1cde-4ac6-a351-79be7fe04a8c\") "
Oct 30 21:06:36.884734 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:36.884648    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d70f6a39-1cde-4ac6-a351-79be7fe04a8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d70f6a39-1cde-4ac6-a351-79be7fe04a8c-maas-api-xks-ca\") pod \"d70f6a39-1cde-4ac6-a351-79be7fe04a8c\" (UID: \"d70f6a39-1cde-4ac6-a351-79be7fe04a8c\") "
Oct 30 21:06:36.884734 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:36.884677    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d70f6a39-1cde-4ac6-a351-79be7fe04a8c-kube-api-access-4f2vn\" (UniqueName: \"kubernetes.io/projected/d70f6a39-1cde-4ac6-a351-79be7fe04a8c-kube-api-access-4f2vn\") pod \"d70f6a39-1cde-4ac6-a351-79be7fe04a8c\" (UID: \"d70f6a39-1cde-4ac6-a351-79be7fe04a8c\") "
Oct 30 21:06:36.885032 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:36.885004    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d70f6a39-1cde-4ac6-a351-79be7fe04a8c-maas-api-openshift-ca" pod "d70f6a39-1cde-4ac6-a351-79be7fe04a8c" (UID: "d70f6a39-1cde-4ac6-a351-79be7fe04a8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:06:36.887031 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:36.887008    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d70f6a39-1cde-4ac6-a351-79be7fe04a8c-kube-api-access-4f2vn" pod "d70f6a39-1cde-4ac6-a351-79be7fe04a8c" (UID: "d70f6a39-1cde-4ac6-a351-79be7fe04a8c"). InnerVolumeSpecName "kube-api-access-4f2vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:36.887106 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:36.887032    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d70f6a39-1cde-4ac6-a351-79be7fe04a8c-maas-api-xks-ca" pod "d70f6a39-1cde-4ac6-a351-79be7fe04a8c" (UID: "d70f6a39-1cde-4ac6-a351-79be7fe04a8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:06:36.985801 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:36.985718    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d70f6a39-1cde-4ac6-a351-79be7fe04a8c-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:36.985801 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:36.985746    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d70f6a39-1cde-4ac6-a351-79be7fe04a8c-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:36.985801 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:36.985756    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f2vn\" (UniqueName: \"kubernetes.io/projected/d70f6a39-1cde-4ac6-a351-79be7fe04a8c-kube-api-access-4f2vn\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:37.043233 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:37.043195    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:06:37.677396 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:37.677368    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j2w5c"
Oct 30 21:06:37.677576 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:37.677407    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j2w5c" event={"ID":"d70f6a39-1cde-4ac6-a351-79be7fe04a8c","Type":"ContainerDied","Data":"d5a9a0379a9075eab004d5f5fafce4d58d9488d5440bda0f5282cfdd8f994d86"}
Oct 30 21:06:37.677576 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:37.677452    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5a9a0379a9075eab004d5f5fafce4d58d9488d5440bda0f5282cfdd8f994d86"
Oct 30 21:06:42.167336 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:42.167294    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:06:44.947015 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:44.946984    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wd946"]
Oct 30 21:06:44.947438 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:44.947427    5071 cpu_manager.go:414] "containerMap: removing container" podUID="d70f6a39-1cde-4ac6-a351-79be7fe04a8c" containerName="revoke-keys"
Oct 30 21:06:44.947501 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:44.947439    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d70f6a39-1cde-4ac6-a351-79be7fe04a8c" containerName="revoke-keys"
Oct 30 21:06:44.947545 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:44.947520    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="d70f6a39-1cde-4ac6-a351-79be7fe04a8c" containerName="revoke-keys"
Oct 30 21:06:44.950748 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:44.950731    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wd946"
Oct 30 21:06:45.009278 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:45.009248    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wd946"]
Oct 30 21:06:45.054334 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:45.054291    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:06:45.054517 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:45.054420    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:06:45.054517 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:45.054420    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 21:06:45.054517 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:45.054448    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:06:45.061879 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:45.061815    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wd946\" (UID: \"bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wd946"
Oct 30 21:06:45.062006 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:45.061966    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wd946\" (UID: \"bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wd946"
Oct 30 21:06:45.062109 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:45.062096    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgcjq\" (UniqueName: \"kubernetes.io/projected/bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014-kube-api-access-dgcjq\") pod \"maas-api-revoke-sub-3d4156f1-wd946\" (UID: \"bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wd946"
Oct 30 21:06:45.163268 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:45.163226    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgcjq\" (UniqueName: \"kubernetes.io/projected/bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014-kube-api-access-dgcjq\") pod \"maas-api-revoke-sub-3d4156f1-wd946\" (UID: \"bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wd946"
Oct 30 21:06:45.163428 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:45.163276    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wd946\" (UID: \"bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wd946"
Oct 30 21:06:45.163428 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:45.163350    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wd946\" (UID: \"bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wd946"
Oct 30 21:06:45.164083 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:45.164066    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wd946\" (UID: \"bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wd946"
Oct 30 21:06:45.166030 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:45.166012    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wd946\" (UID: \"bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wd946"
Oct 30 21:06:45.222647 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:45.222578    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgcjq\" (UniqueName: \"kubernetes.io/projected/bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014-kube-api-access-dgcjq\") pod \"maas-api-revoke-sub-3d4156f1-wd946\" (UID: \"bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wd946"
Oct 30 21:06:45.266488 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:45.266458    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wd946"
Oct 30 21:06:45.278912 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:45.278858    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:06:45.643245 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:45.643220    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wd946"]
Oct 30 21:06:45.644313 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:06:45.644283    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbaf74e0_f8d1_4bbc_bf2c_2f7504d8f014.slice/crio-34bf7a05c561ad098733c611e016842949b4c315a61247332ac54f2d99599705 WatchSource:0}: Error finding container 34bf7a05c561ad098733c611e016842949b4c315a61247332ac54f2d99599705: Status 404 returned error can't find the container with id 34bf7a05c561ad098733c611e016842949b4c315a61247332ac54f2d99599705
Oct 30 21:06:45.722225 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:45.722191    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wd946" event={"ID":"bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014","Type":"ContainerStarted","Data":"34bf7a05c561ad098733c611e016842949b4c315a61247332ac54f2d99599705"}
Oct 30 21:06:46.559038 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:46.559008    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:06:46.727605 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:06:46.727564    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:06:46.730283 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:46.730252    5071 generic.go:356] "Generic (PLEG): container finished" podID="bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014" containerID="17ed1decfff6778053504b688c64dcaef2987e143baa52417821d3b91c3fb4cb" exitCode=0
Oct 30 21:06:46.730402 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:46.730350    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wd946" event={"ID":"bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014","Type":"ContainerDied","Data":"17ed1decfff6778053504b688c64dcaef2987e143baa52417821d3b91c3fb4cb"}
Oct 30 21:06:48.604864 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:48.598590    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wd946"
Oct 30 21:06:48.791370 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:48.791335    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014-maas-api-xks-ca\") pod \"bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014\" (UID: \"bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014\") "
Oct 30 21:06:48.791679 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:48.791654    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014-maas-api-openshift-ca\") pod \"bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014\" (UID: \"bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014\") "
Oct 30 21:06:48.791864 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:48.791846    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014-kube-api-access-dgcjq\" (UniqueName: \"kubernetes.io/projected/bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014-kube-api-access-dgcjq\") pod \"bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014\" (UID: \"bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014\") "
Oct 30 21:06:48.809954 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:48.799518    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014-maas-api-openshift-ca" pod "bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014" (UID: "bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:06:48.826088 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:48.826052    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014-kube-api-access-dgcjq" pod "bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014" (UID: "bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014"). InnerVolumeSpecName "kube-api-access-dgcjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:48.841732 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:48.839945    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014-maas-api-xks-ca" pod "bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014" (UID: "bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:06:48.875859 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:48.866337    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wd946" event={"ID":"bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014","Type":"ContainerDied","Data":"34bf7a05c561ad098733c611e016842949b4c315a61247332ac54f2d99599705"}
Oct 30 21:06:48.875859 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:48.866399    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34bf7a05c561ad098733c611e016842949b4c315a61247332ac54f2d99599705"
Oct 30 21:06:48.875859 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:48.866472    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wd946"
Oct 30 21:06:48.902713 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:48.902445    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgcjq\" (UniqueName: \"kubernetes.io/projected/bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014-kube-api-access-dgcjq\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:48.902713 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:48.902479    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:48.902713 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:48.902493    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:53.053649 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:53.053608    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jwpzr"]
Oct 30 21:06:53.105460 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:53.105420    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-jwpzr"]
Oct 30 21:06:54.200579 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:54.200541    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e53d3b25-1074-434c-a332-00a3b737f173" path="/var/lib/kubelet/pods/e53d3b25-1074-434c-a332-00a3b737f173/volumes"
Oct 30 21:06:59.548342 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:06:59.548297    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:07:02.005851 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:02.002138    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cv4kg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:07:02.005851 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:02.002227    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg" podUID="9be8fbd1-6cb8-4718-8cfe-a45894b5d33e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:07:10.559161 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:10.559125    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:07:16.727690 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:07:16.727654    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:07:18.798838 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:18.798792    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wd946"]
Oct 30 21:07:18.847236 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:18.847196    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:07:18.848811 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:18.848783    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wd946"]
Oct 30 21:07:18.922987 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:18.922950    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mqjl9"]
Oct 30 21:07:18.923435 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:18.923417    5071 cpu_manager.go:414] "containerMap: removing container" podUID="bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014" containerName="revoke-keys"
Oct 30 21:07:18.923435 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:18.923433    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014" containerName="revoke-keys"
Oct 30 21:07:18.923582 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:18.923519    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014" containerName="revoke-keys"
Oct 30 21:07:18.928138 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:18.928114    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mqjl9"
Oct 30 21:07:18.981746 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:18.981722    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:07:18.981946 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:18.981726    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:07:18.981946 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:18.981797    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 21:07:18.981946 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:18.981938    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:07:18.984365 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:18.984316    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mqjl9"]
Oct 30 21:07:19.045473 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:19.045438    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ee9b8f1-7a97-4b10-8dd5-f228652b32e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-mqjl9\" (UID: \"4ee9b8f1-7a97-4b10-8dd5-f228652b32e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mqjl9"
Oct 30 21:07:19.045702 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:19.045500    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w97np\" (UniqueName: \"kubernetes.io/projected/4ee9b8f1-7a97-4b10-8dd5-f228652b32e2-kube-api-access-w97np\") pod \"maas-api-revoke-sub-3d4156f1-mqjl9\" (UID: \"4ee9b8f1-7a97-4b10-8dd5-f228652b32e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mqjl9"
Oct 30 21:07:19.045702 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:19.045613    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ee9b8f1-7a97-4b10-8dd5-f228652b32e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-mqjl9\" (UID: \"4ee9b8f1-7a97-4b10-8dd5-f228652b32e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mqjl9"
Oct 30 21:07:19.146361 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:19.146270    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ee9b8f1-7a97-4b10-8dd5-f228652b32e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-mqjl9\" (UID: \"4ee9b8f1-7a97-4b10-8dd5-f228652b32e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mqjl9"
Oct 30 21:07:19.146361 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:19.146343    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w97np\" (UniqueName: \"kubernetes.io/projected/4ee9b8f1-7a97-4b10-8dd5-f228652b32e2-kube-api-access-w97np\") pod \"maas-api-revoke-sub-3d4156f1-mqjl9\" (UID: \"4ee9b8f1-7a97-4b10-8dd5-f228652b32e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mqjl9"
Oct 30 21:07:19.146609 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:19.146402    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ee9b8f1-7a97-4b10-8dd5-f228652b32e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-mqjl9\" (UID: \"4ee9b8f1-7a97-4b10-8dd5-f228652b32e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mqjl9"
Oct 30 21:07:19.147115 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:19.147092    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ee9b8f1-7a97-4b10-8dd5-f228652b32e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-mqjl9\" (UID: \"4ee9b8f1-7a97-4b10-8dd5-f228652b32e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mqjl9"
Oct 30 21:07:19.149002 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:19.148979    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ee9b8f1-7a97-4b10-8dd5-f228652b32e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-mqjl9\" (UID: \"4ee9b8f1-7a97-4b10-8dd5-f228652b32e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mqjl9"
Oct 30 21:07:19.204460 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:19.204431    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w97np\" (UniqueName: \"kubernetes.io/projected/4ee9b8f1-7a97-4b10-8dd5-f228652b32e2-kube-api-access-w97np\") pod \"maas-api-revoke-sub-3d4156f1-mqjl9\" (UID: \"4ee9b8f1-7a97-4b10-8dd5-f228652b32e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mqjl9"
Oct 30 21:07:19.240958 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:19.240937    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mqjl9"
Oct 30 21:07:19.617759 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:19.617734    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mqjl9"]
Oct 30 21:07:19.619428 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:07:19.619398    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ee9b8f1_7a97_4b10_8dd5_f228652b32e2.slice/crio-2f234b756f9619b66b6b7d03011779453e9c22133d66c58015fd2bb93788c16e WatchSource:0}: Error finding container 2f234b756f9619b66b6b7d03011779453e9c22133d66c58015fd2bb93788c16e: Status 404 returned error can't find the container with id 2f234b756f9619b66b6b7d03011779453e9c22133d66c58015fd2bb93788c16e
Oct 30 21:07:20.092660 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:20.092616    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mqjl9" event={"ID":"4ee9b8f1-7a97-4b10-8dd5-f228652b32e2","Type":"ContainerStarted","Data":"7abf74cb2b45af4d848c83165909547d4b5e9583158bd032c623461ab034b4a9"}
Oct 30 21:07:20.093088 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:20.092670    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mqjl9" event={"ID":"4ee9b8f1-7a97-4b10-8dd5-f228652b32e2","Type":"ContainerStarted","Data":"2f234b756f9619b66b6b7d03011779453e9c22133d66c58015fd2bb93788c16e"}
Oct 30 21:07:20.200234 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:20.200158    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014" path="/var/lib/kubelet/pods/bbaf74e0-f8d1-4bbc-bf2c-2f7504d8f014/volumes"
Oct 30 21:07:20.208831 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:20.208779    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mqjl9" podStartSLOduration=2.20876746 podStartE2EDuration="2.20876746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:07:20.205642268 +0000 UTC m=+2452.122663803" watchObservedRunningTime="2026-10-30 21:07:20.20876746 +0000 UTC m=+2452.125788997"
Oct 30 21:07:21.099421 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:21.099383    5071 generic.go:356] "Generic (PLEG): container finished" podID="4ee9b8f1-7a97-4b10-8dd5-f228652b32e2" containerID="7abf74cb2b45af4d848c83165909547d4b5e9583158bd032c623461ab034b4a9" exitCode=0
Oct 30 21:07:21.099806 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:21.099448    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mqjl9" event={"ID":"4ee9b8f1-7a97-4b10-8dd5-f228652b32e2","Type":"ContainerDied","Data":"7abf74cb2b45af4d848c83165909547d4b5e9583158bd032c623461ab034b4a9"}
Oct 30 21:07:22.253031 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:22.253003    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mqjl9"
Oct 30 21:07:22.382545 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:22.382465    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ee9b8f1-7a97-4b10-8dd5-f228652b32e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ee9b8f1-7a97-4b10-8dd5-f228652b32e2-maas-api-openshift-ca\") pod \"4ee9b8f1-7a97-4b10-8dd5-f228652b32e2\" (UID: \"4ee9b8f1-7a97-4b10-8dd5-f228652b32e2\") "
Oct 30 21:07:22.382545 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:22.382515    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ee9b8f1-7a97-4b10-8dd5-f228652b32e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ee9b8f1-7a97-4b10-8dd5-f228652b32e2-maas-api-xks-ca\") pod \"4ee9b8f1-7a97-4b10-8dd5-f228652b32e2\" (UID: \"4ee9b8f1-7a97-4b10-8dd5-f228652b32e2\") "
Oct 30 21:07:22.382812 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:22.382554    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ee9b8f1-7a97-4b10-8dd5-f228652b32e2-kube-api-access-w97np\" (UniqueName: \"kubernetes.io/projected/4ee9b8f1-7a97-4b10-8dd5-f228652b32e2-kube-api-access-w97np\") pod \"4ee9b8f1-7a97-4b10-8dd5-f228652b32e2\" (UID: \"4ee9b8f1-7a97-4b10-8dd5-f228652b32e2\") "
Oct 30 21:07:22.382925 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:22.382898    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ee9b8f1-7a97-4b10-8dd5-f228652b32e2-maas-api-openshift-ca" pod "4ee9b8f1-7a97-4b10-8dd5-f228652b32e2" (UID: "4ee9b8f1-7a97-4b10-8dd5-f228652b32e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:07:22.384887 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:22.384858    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ee9b8f1-7a97-4b10-8dd5-f228652b32e2-maas-api-xks-ca" pod "4ee9b8f1-7a97-4b10-8dd5-f228652b32e2" (UID: "4ee9b8f1-7a97-4b10-8dd5-f228652b32e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:07:22.384994 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:22.384973    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ee9b8f1-7a97-4b10-8dd5-f228652b32e2-kube-api-access-w97np" pod "4ee9b8f1-7a97-4b10-8dd5-f228652b32e2" (UID: "4ee9b8f1-7a97-4b10-8dd5-f228652b32e2"). InnerVolumeSpecName "kube-api-access-w97np". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:07:22.483443 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:22.483406    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ee9b8f1-7a97-4b10-8dd5-f228652b32e2-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:07:22.483443 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:22.483437    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ee9b8f1-7a97-4b10-8dd5-f228652b32e2-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:07:22.483626 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:22.483451    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w97np\" (UniqueName: \"kubernetes.io/projected/4ee9b8f1-7a97-4b10-8dd5-f228652b32e2-kube-api-access-w97np\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:07:23.112223 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:23.112184    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mqjl9" event={"ID":"4ee9b8f1-7a97-4b10-8dd5-f228652b32e2","Type":"ContainerDied","Data":"2f234b756f9619b66b6b7d03011779453e9c22133d66c58015fd2bb93788c16e"}
Oct 30 21:07:23.112223 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:23.112220    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mqjl9"
Oct 30 21:07:23.112451 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:23.112243    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f234b756f9619b66b6b7d03011779453e9c22133d66c58015fd2bb93788c16e"
Oct 30 21:07:33.555051 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:33.555012    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:07:34.050428 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:34.050397    5071 scope.go:122] "RemoveContainer" containerID="961e19dd805717aa5fcc892ab6781e7b1c30499572da7915a6910274cd947f34"
Oct 30 21:07:34.060252 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:34.060228    5071 scope.go:122] "RemoveContainer" containerID="598e1dea3d58f6d8f42acb85156d12cb6729954811abb8084dd1efff6b424ddc"
Oct 30 21:07:36.624171 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:36.624126    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:07:42.451313 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:42.451198    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:07:43.783316 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:43.783279    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:07:45.608520 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:45.608481    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:07:46.727544 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:07:46.727497    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:07:53.060254 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:53.060206    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:07:53.257536 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:53.257495    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:07:57.234707 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:07:57.234674    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:08:06.051490 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:06.051452    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xz7s7"]
Oct 30 21:08:06.106867 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:06.106812    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-xz7s7"]
Oct 30 21:08:06.121082 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:06.121051    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pdgs9"]
Oct 30 21:08:06.121502 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:06.121488    5071 cpu_manager.go:414] "containerMap: removing container" podUID="4ee9b8f1-7a97-4b10-8dd5-f228652b32e2" containerName="revoke-keys"
Oct 30 21:08:06.121546 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:06.121504    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ee9b8f1-7a97-4b10-8dd5-f228652b32e2" containerName="revoke-keys"
Oct 30 21:08:06.121609 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:06.121600    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ee9b8f1-7a97-4b10-8dd5-f228652b32e2" containerName="revoke-keys"
Oct 30 21:08:06.125125 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:06.125108    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pdgs9"
Oct 30 21:08:06.180719 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:06.180690    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:08:06.180974 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:06.180932    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:08:06.181085 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:06.180960    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 21:08:06.181174 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:06.180967    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:08:06.183256 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:06.183232    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pdgs9"]
Oct 30 21:08:06.201259 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:06.201231    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2be78af-3d58-44af-afe4-0a684ed7a7e5" path="/var/lib/kubelet/pods/a2be78af-3d58-44af-afe4-0a684ed7a7e5/volumes"
Oct 30 21:08:06.234579 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:06.234546    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhrl5\" (UniqueName: \"kubernetes.io/projected/7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c-kube-api-access-jhrl5\") pod \"maas-api-revoke-sub-e2fc9c5b-pdgs9\" (UID: \"7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pdgs9"
Oct 30 21:08:06.234734 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:06.234685    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-pdgs9\" (UID: \"7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pdgs9"
Oct 30 21:08:06.234734 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:06.234730    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-pdgs9\" (UID: \"7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pdgs9"
Oct 30 21:08:06.335879 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:06.335791    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhrl5\" (UniqueName: \"kubernetes.io/projected/7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c-kube-api-access-jhrl5\") pod \"maas-api-revoke-sub-e2fc9c5b-pdgs9\" (UID: \"7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pdgs9"
Oct 30 21:08:06.336056 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:06.335918    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-pdgs9\" (UID: \"7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pdgs9"
Oct 30 21:08:06.336056 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:06.335970    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-pdgs9\" (UID: \"7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pdgs9"
Oct 30 21:08:06.336685 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:06.336661    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-pdgs9\" (UID: \"7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pdgs9"
Oct 30 21:08:06.338546 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:06.338522    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-pdgs9\" (UID: \"7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pdgs9"
Oct 30 21:08:06.394043 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:06.394014    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhrl5\" (UniqueName: \"kubernetes.io/projected/7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c-kube-api-access-jhrl5\") pod \"maas-api-revoke-sub-e2fc9c5b-pdgs9\" (UID: \"7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pdgs9"
Oct 30 21:08:06.441174 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:06.441143    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pdgs9"
Oct 30 21:08:06.813740 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:06.813712    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pdgs9"]
Oct 30 21:08:06.815270 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:08:06.815244    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a9e0b21_0a11_4046_ba1e_2a5631c5ae8c.slice/crio-ca030a17ae2fcb7989317707c7de4fbd9bb6ab76e96a0ce513869fa35543ce2d WatchSource:0}: Error finding container ca030a17ae2fcb7989317707c7de4fbd9bb6ab76e96a0ce513869fa35543ce2d: Status 404 returned error can't find the container with id ca030a17ae2fcb7989317707c7de4fbd9bb6ab76e96a0ce513869fa35543ce2d
Oct 30 21:08:07.401182 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:07.401088    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pdgs9" event={"ID":"7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c","Type":"ContainerStarted","Data":"f50ece802a2a2397731254ce77a2072b104d40be7ab81b51915b49ecf4b8f3f0"}
Oct 30 21:08:07.401182 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:07.401156    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pdgs9" event={"ID":"7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c","Type":"ContainerStarted","Data":"ca030a17ae2fcb7989317707c7de4fbd9bb6ab76e96a0ce513869fa35543ce2d"}
Oct 30 21:08:07.518608 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:07.518545    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pdgs9" podStartSLOduration=1.5185321489999999 podStartE2EDuration="1.518532149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:07.51599097 +0000 UTC m=+2499.433012507" watchObservedRunningTime="2026-10-30 21:08:07.518532149 +0000 UTC m=+2499.435553685"
Oct 30 21:08:08.408566 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:08.408529    5071 generic.go:356] "Generic (PLEG): container finished" podID="7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c" containerID="f50ece802a2a2397731254ce77a2072b104d40be7ab81b51915b49ecf4b8f3f0" exitCode=0
Oct 30 21:08:08.409011 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:08.408608    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pdgs9" event={"ID":"7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c","Type":"ContainerDied","Data":"f50ece802a2a2397731254ce77a2072b104d40be7ab81b51915b49ecf4b8f3f0"}
Oct 30 21:08:09.555111 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:09.555082    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pdgs9"
Oct 30 21:08:09.671360 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:09.671269    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c-kube-api-access-jhrl5\" (UniqueName: \"kubernetes.io/projected/7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c-kube-api-access-jhrl5\") pod \"7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c\" (UID: \"7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c\") "
Oct 30 21:08:09.671360 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:09.671313    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c-maas-api-xks-ca\") pod \"7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c\" (UID: \"7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c\") "
Oct 30 21:08:09.671608 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:09.671450    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c-maas-api-openshift-ca\") pod \"7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c\" (UID: \"7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c\") "
Oct 30 21:08:09.672091 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:09.672048    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c-maas-api-openshift-ca" pod "7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c" (UID: "7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:08:09.674332 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:09.674281    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c-maas-api-xks-ca" pod "7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c" (UID: "7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:08:09.674723 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:09.674698    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c-kube-api-access-jhrl5" pod "7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c" (UID: "7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c"). InnerVolumeSpecName "kube-api-access-jhrl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:08:09.772973 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:09.772943    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhrl5\" (UniqueName: \"kubernetes.io/projected/7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c-kube-api-access-jhrl5\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:09.772973 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:09.772970    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:09.773135 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:09.772980    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:10.421744 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:10.421714    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pdgs9"
Oct 30 21:08:10.421970 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:10.421763    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pdgs9" event={"ID":"7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c","Type":"ContainerDied","Data":"ca030a17ae2fcb7989317707c7de4fbd9bb6ab76e96a0ce513869fa35543ce2d"}
Oct 30 21:08:10.421970 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:10.421816    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca030a17ae2fcb7989317707c7de4fbd9bb6ab76e96a0ce513869fa35543ce2d"
Oct 30 21:08:16.727272 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:08:16.727228    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:08:16.874050 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:16.874019    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v9xhv"]
Oct 30 21:08:16.874522 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:16.874508    5071 cpu_manager.go:414] "containerMap: removing container" podUID="7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c" containerName="revoke-keys"
Oct 30 21:08:16.874568 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:16.874524    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c" containerName="revoke-keys"
Oct 30 21:08:16.874619 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:16.874608    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c" containerName="revoke-keys"
Oct 30 21:08:16.878081 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:16.878060    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v9xhv"
Oct 30 21:08:16.936507 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:16.936476    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v9xhv"]
Oct 30 21:08:16.982743 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:16.982656    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:08:16.982743 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:16.982655    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:08:16.982743 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:16.982739    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 21:08:16.982998 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:16.982755    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:08:17.042406 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:17.042375    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nsdr\" (UniqueName: \"kubernetes.io/projected/2c764024-4a55-4350-92df-03d5bf116fe7-kube-api-access-9nsdr\") pod \"maas-api-revoke-sub-5a29efb3-v9xhv\" (UID: \"2c764024-4a55-4350-92df-03d5bf116fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v9xhv"
Oct 30 21:08:17.042560 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:17.042430    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c764024-4a55-4350-92df-03d5bf116fe7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-v9xhv\" (UID: \"2c764024-4a55-4350-92df-03d5bf116fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v9xhv"
Oct 30 21:08:17.042560 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:17.042499    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c764024-4a55-4350-92df-03d5bf116fe7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-v9xhv\" (UID: \"2c764024-4a55-4350-92df-03d5bf116fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v9xhv"
Oct 30 21:08:17.143598 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:17.143567    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c764024-4a55-4350-92df-03d5bf116fe7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-v9xhv\" (UID: \"2c764024-4a55-4350-92df-03d5bf116fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v9xhv"
Oct 30 21:08:17.143764 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:17.143663    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nsdr\" (UniqueName: \"kubernetes.io/projected/2c764024-4a55-4350-92df-03d5bf116fe7-kube-api-access-9nsdr\") pod \"maas-api-revoke-sub-5a29efb3-v9xhv\" (UID: \"2c764024-4a55-4350-92df-03d5bf116fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v9xhv"
Oct 30 21:08:17.143764 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:17.143702    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c764024-4a55-4350-92df-03d5bf116fe7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-v9xhv\" (UID: \"2c764024-4a55-4350-92df-03d5bf116fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v9xhv"
Oct 30 21:08:17.144287 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:17.144267    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c764024-4a55-4350-92df-03d5bf116fe7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-v9xhv\" (UID: \"2c764024-4a55-4350-92df-03d5bf116fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v9xhv"
Oct 30 21:08:17.146124 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:17.146103    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c764024-4a55-4350-92df-03d5bf116fe7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-v9xhv\" (UID: \"2c764024-4a55-4350-92df-03d5bf116fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v9xhv"
Oct 30 21:08:17.202402 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:17.202370    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nsdr\" (UniqueName: \"kubernetes.io/projected/2c764024-4a55-4350-92df-03d5bf116fe7-kube-api-access-9nsdr\") pod \"maas-api-revoke-sub-5a29efb3-v9xhv\" (UID: \"2c764024-4a55-4350-92df-03d5bf116fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v9xhv"
Oct 30 21:08:17.490699 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:17.490644    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v9xhv"
Oct 30 21:08:17.864505 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:17.864474    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v9xhv"]
Oct 30 21:08:17.866539 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:08:17.866513    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c764024_4a55_4350_92df_03d5bf116fe7.slice/crio-6438cbb908ae3bdba669ace40dd737747358c8d6e5db31083b7244fb3fbc2618 WatchSource:0}: Error finding container 6438cbb908ae3bdba669ace40dd737747358c8d6e5db31083b7244fb3fbc2618: Status 404 returned error can't find the container with id 6438cbb908ae3bdba669ace40dd737747358c8d6e5db31083b7244fb3fbc2618
Oct 30 21:08:18.469341 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:18.469246    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v9xhv" event={"ID":"2c764024-4a55-4350-92df-03d5bf116fe7","Type":"ContainerStarted","Data":"5cc87a33fac2c3460604d86b01ba09221937e9c9df20bf5ea191bca0c6a68866"}
Oct 30 21:08:18.469341 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:18.469313    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v9xhv" event={"ID":"2c764024-4a55-4350-92df-03d5bf116fe7","Type":"ContainerStarted","Data":"6438cbb908ae3bdba669ace40dd737747358c8d6e5db31083b7244fb3fbc2618"}
Oct 30 21:08:18.587049 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:18.586992    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v9xhv" podStartSLOduration=2.5869776 podStartE2EDuration="2.5869776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:18.583663795 +0000 UTC m=+2510.500685328" watchObservedRunningTime="2026-10-30 21:08:18.5869776 +0000 UTC m=+2510.503999127"
Oct 30 21:08:19.147996 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:19.147955    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:08:19.479122 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:19.479026    5071 generic.go:356] "Generic (PLEG): container finished" podID="2c764024-4a55-4350-92df-03d5bf116fe7" containerID="5cc87a33fac2c3460604d86b01ba09221937e9c9df20bf5ea191bca0c6a68866" exitCode=0
Oct 30 21:08:19.479313 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:19.479202    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v9xhv" event={"ID":"2c764024-4a55-4350-92df-03d5bf116fe7","Type":"ContainerDied","Data":"5cc87a33fac2c3460604d86b01ba09221937e9c9df20bf5ea191bca0c6a68866"}
Oct 30 21:08:21.494847 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:21.494426    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v9xhv"
Oct 30 21:08:21.533846 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:21.532926    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v9xhv" event={"ID":"2c764024-4a55-4350-92df-03d5bf116fe7","Type":"ContainerDied","Data":"6438cbb908ae3bdba669ace40dd737747358c8d6e5db31083b7244fb3fbc2618"}
Oct 30 21:08:21.533846 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:21.532987    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6438cbb908ae3bdba669ace40dd737747358c8d6e5db31083b7244fb3fbc2618"
Oct 30 21:08:21.533846 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:21.533077    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v9xhv"
Oct 30 21:08:21.567844 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:21.559730    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c764024-4a55-4350-92df-03d5bf116fe7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c764024-4a55-4350-92df-03d5bf116fe7-maas-api-xks-ca\") pod \"2c764024-4a55-4350-92df-03d5bf116fe7\" (UID: \"2c764024-4a55-4350-92df-03d5bf116fe7\") "
Oct 30 21:08:21.567844 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:21.559949    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c764024-4a55-4350-92df-03d5bf116fe7-kube-api-access-9nsdr\" (UniqueName: \"kubernetes.io/projected/2c764024-4a55-4350-92df-03d5bf116fe7-kube-api-access-9nsdr\") pod \"2c764024-4a55-4350-92df-03d5bf116fe7\" (UID: \"2c764024-4a55-4350-92df-03d5bf116fe7\") "
Oct 30 21:08:21.567844 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:21.560011    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c764024-4a55-4350-92df-03d5bf116fe7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c764024-4a55-4350-92df-03d5bf116fe7-maas-api-openshift-ca\") pod \"2c764024-4a55-4350-92df-03d5bf116fe7\" (UID: \"2c764024-4a55-4350-92df-03d5bf116fe7\") "
Oct 30 21:08:21.567844 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:21.560757    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c764024-4a55-4350-92df-03d5bf116fe7-maas-api-openshift-ca" pod "2c764024-4a55-4350-92df-03d5bf116fe7" (UID: "2c764024-4a55-4350-92df-03d5bf116fe7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:08:21.570491 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:21.570466    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c764024-4a55-4350-92df-03d5bf116fe7-kube-api-access-9nsdr" pod "2c764024-4a55-4350-92df-03d5bf116fe7" (UID: "2c764024-4a55-4350-92df-03d5bf116fe7"). InnerVolumeSpecName "kube-api-access-9nsdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:08:21.581418 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:21.581393    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c764024-4a55-4350-92df-03d5bf116fe7-maas-api-xks-ca" pod "2c764024-4a55-4350-92df-03d5bf116fe7" (UID: "2c764024-4a55-4350-92df-03d5bf116fe7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:08:21.644959 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:21.644925    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-xhg2f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:08:21.645226 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:21.645197    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xhg2f" podUID="562c4391-03f9-470d-a007-b4931bd3f421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:08:21.668902 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:21.661447    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nsdr\" (UniqueName: \"kubernetes.io/projected/2c764024-4a55-4350-92df-03d5bf116fe7-kube-api-access-9nsdr\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:21.669094 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:21.669071    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c764024-4a55-4350-92df-03d5bf116fe7-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:21.669207 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:21.669194    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c764024-4a55-4350-92df-03d5bf116fe7-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:21.682941 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:21.682909    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-pmwpk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:08:21.683134 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:21.683110    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-pmwpk" podUID="a367ee40-c5ea-44f3-92c6-d1b5cf1141b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:08:29.966497 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:29.966460    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:08:33.982425 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:33.982387    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mqjl9"]
Oct 30 21:08:34.033517 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:34.033476    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mqjl9"]
Oct 30 21:08:34.090097 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:34.090064    5071 scope.go:122] "RemoveContainer" containerID="83a374a482d0674209c062d1052135eaeb5b3f370c4bea3ad745a1085cd1bb31"
Oct 30 21:08:34.105164 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:34.105139    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g78dg"]
Oct 30 21:08:34.105795 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:34.105775    5071 cpu_manager.go:414] "containerMap: removing container" podUID="2c764024-4a55-4350-92df-03d5bf116fe7" containerName="revoke-keys"
Oct 30 21:08:34.105868 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:34.105801    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c764024-4a55-4350-92df-03d5bf116fe7" containerName="revoke-keys"
Oct 30 21:08:34.106001 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:34.105985    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c764024-4a55-4350-92df-03d5bf116fe7" containerName="revoke-keys"
Oct 30 21:08:34.110780 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:34.110763    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g78dg"
Oct 30 21:08:34.163884 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:34.163815    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61fa5afd-c506-48bc-9166-93b0bc94cf57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-g78dg\" (UID: \"61fa5afd-c506-48bc-9166-93b0bc94cf57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g78dg"
Oct 30 21:08:34.164075 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:34.163957    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61fa5afd-c506-48bc-9166-93b0bc94cf57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-g78dg\" (UID: \"61fa5afd-c506-48bc-9166-93b0bc94cf57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g78dg"
Oct 30 21:08:34.164075 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:34.163980    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52nzz\" (UniqueName: \"kubernetes.io/projected/61fa5afd-c506-48bc-9166-93b0bc94cf57-kube-api-access-52nzz\") pod \"maas-api-revoke-sub-3d4156f1-g78dg\" (UID: \"61fa5afd-c506-48bc-9166-93b0bc94cf57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g78dg"
Oct 30 21:08:34.164791 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:34.164763    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:08:34.168349 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:34.164880    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:08:34.168349 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:34.164934    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:08:34.168349 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:34.165008    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 21:08:34.168349 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:34.167230    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g78dg"]
Oct 30 21:08:34.201325 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:34.201293    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ee9b8f1-7a97-4b10-8dd5-f228652b32e2" path="/var/lib/kubelet/pods/4ee9b8f1-7a97-4b10-8dd5-f228652b32e2/volumes"
Oct 30 21:08:34.265237 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:34.265156    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61fa5afd-c506-48bc-9166-93b0bc94cf57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-g78dg\" (UID: \"61fa5afd-c506-48bc-9166-93b0bc94cf57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g78dg"
Oct 30 21:08:34.265237 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:34.265194    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52nzz\" (UniqueName: \"kubernetes.io/projected/61fa5afd-c506-48bc-9166-93b0bc94cf57-kube-api-access-52nzz\") pod \"maas-api-revoke-sub-3d4156f1-g78dg\" (UID: \"61fa5afd-c506-48bc-9166-93b0bc94cf57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g78dg"
Oct 30 21:08:34.265433 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:34.265253    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61fa5afd-c506-48bc-9166-93b0bc94cf57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-g78dg\" (UID: \"61fa5afd-c506-48bc-9166-93b0bc94cf57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g78dg"
Oct 30 21:08:34.266043 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:34.266015    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61fa5afd-c506-48bc-9166-93b0bc94cf57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-g78dg\" (UID: \"61fa5afd-c506-48bc-9166-93b0bc94cf57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g78dg"
Oct 30 21:08:34.267772 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:34.267741    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61fa5afd-c506-48bc-9166-93b0bc94cf57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-g78dg\" (UID: \"61fa5afd-c506-48bc-9166-93b0bc94cf57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g78dg"
Oct 30 21:08:34.323909 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:34.323878    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52nzz\" (UniqueName: \"kubernetes.io/projected/61fa5afd-c506-48bc-9166-93b0bc94cf57-kube-api-access-52nzz\") pod \"maas-api-revoke-sub-3d4156f1-g78dg\" (UID: \"61fa5afd-c506-48bc-9166-93b0bc94cf57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g78dg"
Oct 30 21:08:34.423555 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:34.423517    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g78dg"
Oct 30 21:08:34.798038 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:34.798012    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g78dg"]
Oct 30 21:08:34.799467 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:08:34.799434    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61fa5afd_c506_48bc_9166_93b0bc94cf57.slice/crio-0b6137292d5686335b390cc38d3aebd7facf8163caa94161288234110d83461d WatchSource:0}: Error finding container 0b6137292d5686335b390cc38d3aebd7facf8163caa94161288234110d83461d: Status 404 returned error can't find the container with id 0b6137292d5686335b390cc38d3aebd7facf8163caa94161288234110d83461d
Oct 30 21:08:35.613846 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:35.613780    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g78dg" event={"ID":"61fa5afd-c506-48bc-9166-93b0bc94cf57","Type":"ContainerStarted","Data":"9df0582b16cf767056936d304feaad03013e70abd13d0d859543f0efc505bd1e"}
Oct 30 21:08:35.614235 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:35.613852    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g78dg" event={"ID":"61fa5afd-c506-48bc-9166-93b0bc94cf57","Type":"ContainerStarted","Data":"0b6137292d5686335b390cc38d3aebd7facf8163caa94161288234110d83461d"}
Oct 30 21:08:35.727846 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:35.727770    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g78dg" podStartSLOduration=1.727753441 podStartE2EDuration="1.727753441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:35.726934877 +0000 UTC m=+2527.643956413" watchObservedRunningTime="2026-10-30 21:08:35.727753441 +0000 UTC m=+2527.644774977"
Oct 30 21:08:36.620808 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:36.620769    5071 generic.go:356] "Generic (PLEG): container finished" podID="61fa5afd-c506-48bc-9166-93b0bc94cf57" containerID="9df0582b16cf767056936d304feaad03013e70abd13d0d859543f0efc505bd1e" exitCode=0
Oct 30 21:08:36.621240 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:36.620866    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g78dg" event={"ID":"61fa5afd-c506-48bc-9166-93b0bc94cf57","Type":"ContainerDied","Data":"9df0582b16cf767056936d304feaad03013e70abd13d0d859543f0efc505bd1e"}
Oct 30 21:08:36.980967 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:36.980881    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:08:37.767810 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:37.767784    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g78dg"
Oct 30 21:08:37.908543 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:37.908470    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61fa5afd-c506-48bc-9166-93b0bc94cf57-kube-api-access-52nzz\" (UniqueName: \"kubernetes.io/projected/61fa5afd-c506-48bc-9166-93b0bc94cf57-kube-api-access-52nzz\") pod \"61fa5afd-c506-48bc-9166-93b0bc94cf57\" (UID: \"61fa5afd-c506-48bc-9166-93b0bc94cf57\") "
Oct 30 21:08:37.908704 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:37.908547    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61fa5afd-c506-48bc-9166-93b0bc94cf57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61fa5afd-c506-48bc-9166-93b0bc94cf57-maas-api-xks-ca\") pod \"61fa5afd-c506-48bc-9166-93b0bc94cf57\" (UID: \"61fa5afd-c506-48bc-9166-93b0bc94cf57\") "
Oct 30 21:08:37.908704 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:37.908618    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61fa5afd-c506-48bc-9166-93b0bc94cf57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61fa5afd-c506-48bc-9166-93b0bc94cf57-maas-api-openshift-ca\") pod \"61fa5afd-c506-48bc-9166-93b0bc94cf57\" (UID: \"61fa5afd-c506-48bc-9166-93b0bc94cf57\") "
Oct 30 21:08:37.909060 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:37.909031    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61fa5afd-c506-48bc-9166-93b0bc94cf57-maas-api-openshift-ca" pod "61fa5afd-c506-48bc-9166-93b0bc94cf57" (UID: "61fa5afd-c506-48bc-9166-93b0bc94cf57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:08:37.910931 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:37.910900    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61fa5afd-c506-48bc-9166-93b0bc94cf57-kube-api-access-52nzz" pod "61fa5afd-c506-48bc-9166-93b0bc94cf57" (UID: "61fa5afd-c506-48bc-9166-93b0bc94cf57"). InnerVolumeSpecName "kube-api-access-52nzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:08:37.911355 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:37.911332    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61fa5afd-c506-48bc-9166-93b0bc94cf57-maas-api-xks-ca" pod "61fa5afd-c506-48bc-9166-93b0bc94cf57" (UID: "61fa5afd-c506-48bc-9166-93b0bc94cf57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:08:38.009866 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:38.009836    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61fa5afd-c506-48bc-9166-93b0bc94cf57-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:38.009866 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:38.009864    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52nzz\" (UniqueName: \"kubernetes.io/projected/61fa5afd-c506-48bc-9166-93b0bc94cf57-kube-api-access-52nzz\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:38.010068 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:38.009875    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61fa5afd-c506-48bc-9166-93b0bc94cf57-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:38.051542 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:38.051509    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hg58l"]
Oct 30 21:08:38.055510 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:38.055485    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hg58l"]
Oct 30 21:08:38.201492 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:38.201408    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d35530f-1a3b-485d-a41f-dd20fbcb5b01" path="/var/lib/kubelet/pods/2d35530f-1a3b-485d-a41f-dd20fbcb5b01/volumes"
Oct 30 21:08:38.633760 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:38.633728    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g78dg"
Oct 30 21:08:38.633965 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:38.633779    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g78dg" event={"ID":"61fa5afd-c506-48bc-9166-93b0bc94cf57","Type":"ContainerDied","Data":"0b6137292d5686335b390cc38d3aebd7facf8163caa94161288234110d83461d"}
Oct 30 21:08:38.633965 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:38.633842    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b6137292d5686335b390cc38d3aebd7facf8163caa94161288234110d83461d"
Oct 30 21:08:42.810844 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:42.810784    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:08:45.945743 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:45.945709    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:08:46.727055 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:08:46.727020    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:08:48.694837 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:48.694790    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:08:50.089539 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:50.089462    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clfv7"]
Oct 30 21:08:50.109356 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:50.109320    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-clfv7"]
Oct 30 21:08:50.249854 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:50.249076    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67" path="/var/lib/kubelet/pods/d1c0c6eb-9df2-4c5f-84cd-d701bf7c3d67/volumes"
Oct 30 21:08:54.644160 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:54.644115    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:08:59.541807 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:08:59.541769    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:09:06.145804 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:06.145761    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:06.632448 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:06.632409    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"]
Oct 30 21:09:06.632909 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:06.632896    5071 cpu_manager.go:414] "containerMap: removing container" podUID="61fa5afd-c506-48bc-9166-93b0bc94cf57" containerName="revoke-keys"
Oct 30 21:09:06.632956 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:06.632911    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61fa5afd-c506-48bc-9166-93b0bc94cf57" containerName="revoke-keys"
Oct 30 21:09:06.632996 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:06.632990    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="61fa5afd-c506-48bc-9166-93b0bc94cf57" containerName="revoke-keys"
Oct 30 21:09:06.636283 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:06.636267    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:09:06.699554 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:06.699482    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-7919c9-openshift-default-dockercfg-gwzmj\""
Oct 30 21:09:06.749909 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:06.749877    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"]
Oct 30 21:09:06.769449 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:06.769415    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:06.771681 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:06.771648    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be86287d-e81f-4b44-8219-c64fc1c56288-istio-podinfo\") pod \"e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7\" (UID: \"be86287d-e81f-4b44-8219-c64fc1c56288\") " pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:09:06.771786 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:06.771697    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be86287d-e81f-4b44-8219-c64fc1c56288-istio-token\") pod \"e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7\" (UID: \"be86287d-e81f-4b44-8219-c64fc1c56288\") " pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:09:06.771866 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:06.771805    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be86287d-e81f-4b44-8219-c64fc1c56288-istiod-ca-cert\") pod \"e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7\" (UID: \"be86287d-e81f-4b44-8219-c64fc1c56288\") " pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:09:06.771915 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:06.771892    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be86287d-e81f-4b44-8219-c64fc1c56288-istio-data\") pod \"e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7\" (UID: \"be86287d-e81f-4b44-8219-c64fc1c56288\") " pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:09:06.771979 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:06.771960    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj6kq\" (UniqueName: \"kubernetes.io/projected/be86287d-e81f-4b44-8219-c64fc1c56288-kube-api-access-mj6kq\") pod \"e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7\" (UID: \"be86287d-e81f-4b44-8219-c64fc1c56288\") " pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:09:06.772029 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:06.772016    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be86287d-e81f-4b44-8219-c64fc1c56288-credential-socket\") pod \"e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7\" (UID: \"be86287d-e81f-4b44-8219-c64fc1c56288\") " pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:09:06.772101 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:06.772082    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be86287d-e81f-4b44-8219-c64fc1c56288-istio-envoy\") pod \"e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7\" (UID: \"be86287d-e81f-4b44-8219-c64fc1c56288\") " pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:09:06.772144 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:06.772131    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be86287d-e81f-4b44-8219-c64fc1c56288-workload-certs\") pod \"e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7\" (UID: \"be86287d-e81f-4b44-8219-c64fc1c56288\") " pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:09:06.772207 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:06.772192    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be86287d-e81f-4b44-8219-c64fc1c56288-workload-socket\") pod \"e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7\" (UID: \"be86287d-e81f-4b44-8219-c64fc1c56288\") " pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:09:06.872732 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:06.872698    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be86287d-e81f-4b44-8219-c64fc1c56288-workload-socket\") pod \"e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7\" (UID: \"be86287d-e81f-4b44-8219-c64fc1c56288\") " pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:09:06.872946 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:06.872745    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be86287d-e81f-4b44-8219-c64fc1c56288-istio-podinfo\") pod \"e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7\" (UID: \"be86287d-e81f-4b44-8219-c64fc1c56288\") " pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:09:06.872946 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:06.872762    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be86287d-e81f-4b44-8219-c64fc1c56288-istio-token\") pod \"e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7\" (UID: \"be86287d-e81f-4b44-8219-c64fc1c56288\") " pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:09:06.872946 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:06.872790    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be86287d-e81f-4b44-8219-c64fc1c56288-istiod-ca-cert\") pod \"e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7\" (UID: \"be86287d-e81f-4b44-8219-c64fc1c56288\") " pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:09:06.872946 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:06.872813    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be86287d-e81f-4b44-8219-c64fc1c56288-istio-data\") pod \"e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7\" (UID: \"be86287d-e81f-4b44-8219-c64fc1c56288\") " pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:09:06.872946 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:06.872886    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj6kq\" (UniqueName: \"kubernetes.io/projected/be86287d-e81f-4b44-8219-c64fc1c56288-kube-api-access-mj6kq\") pod \"e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7\" (UID: \"be86287d-e81f-4b44-8219-c64fc1c56288\") " pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:09:06.872946 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:06.872920    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be86287d-e81f-4b44-8219-c64fc1c56288-credential-socket\") pod \"e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7\" (UID: \"be86287d-e81f-4b44-8219-c64fc1c56288\") " pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:09:06.873304 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:06.872974    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be86287d-e81f-4b44-8219-c64fc1c56288-istio-envoy\") pod \"e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7\" (UID: \"be86287d-e81f-4b44-8219-c64fc1c56288\") " pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:09:06.873304 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:06.873066    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be86287d-e81f-4b44-8219-c64fc1c56288-workload-certs\") pod \"e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7\" (UID: \"be86287d-e81f-4b44-8219-c64fc1c56288\") " pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:09:06.873304 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:06.873289    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be86287d-e81f-4b44-8219-c64fc1c56288-istio-data\") pod \"e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7\" (UID: \"be86287d-e81f-4b44-8219-c64fc1c56288\") " pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:09:06.873475 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:06.873332    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be86287d-e81f-4b44-8219-c64fc1c56288-credential-socket\") pod \"e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7\" (UID: \"be86287d-e81f-4b44-8219-c64fc1c56288\") " pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:09:06.873475 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:06.873399    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be86287d-e81f-4b44-8219-c64fc1c56288-workload-certs\") pod \"e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7\" (UID: \"be86287d-e81f-4b44-8219-c64fc1c56288\") " pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:09:06.873475 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:06.873433    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be86287d-e81f-4b44-8219-c64fc1c56288-workload-socket\") pod \"e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7\" (UID: \"be86287d-e81f-4b44-8219-c64fc1c56288\") " pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:09:06.873948 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:06.873923    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be86287d-e81f-4b44-8219-c64fc1c56288-istiod-ca-cert\") pod \"e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7\" (UID: \"be86287d-e81f-4b44-8219-c64fc1c56288\") " pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:09:06.875496 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:06.875473    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be86287d-e81f-4b44-8219-c64fc1c56288-istio-envoy\") pod \"e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7\" (UID: \"be86287d-e81f-4b44-8219-c64fc1c56288\") " pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:09:06.875951 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:06.875927    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be86287d-e81f-4b44-8219-c64fc1c56288-istio-podinfo\") pod \"e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7\" (UID: \"be86287d-e81f-4b44-8219-c64fc1c56288\") " pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:09:06.933321 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:06.933160    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be86287d-e81f-4b44-8219-c64fc1c56288-istio-token\") pod \"e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7\" (UID: \"be86287d-e81f-4b44-8219-c64fc1c56288\") " pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:09:06.934926 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:06.934905    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj6kq\" (UniqueName: \"kubernetes.io/projected/be86287d-e81f-4b44-8219-c64fc1c56288-kube-api-access-mj6kq\") pod \"e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7\" (UID: \"be86287d-e81f-4b44-8219-c64fc1c56288\") " pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:09:06.949735 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:06.949712    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:09:07.159059 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:07.158973    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:07.381392 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:07.381364    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"]
Oct 30 21:09:07.382635 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:09:07.382600    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe86287d_e81f_4b44_8219_c64fc1c56288.slice/crio-5aa206e0cddc7f36f8f2b4f8e7691dd44a11971cf9ead2d792761621f6fb98ab WatchSource:0}: Error finding container 5aa206e0cddc7f36f8f2b4f8e7691dd44a11971cf9ead2d792761621f6fb98ab: Status 404 returned error can't find the container with id 5aa206e0cddc7f36f8f2b4f8e7691dd44a11971cf9ead2d792761621f6fb98ab
Oct 30 21:09:07.384657 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:07.384625    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:09:07.384728 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:07.384690    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:09:07.384728 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:07.384721    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:09:07.848902 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:07.848864    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7" event={"ID":"be86287d-e81f-4b44-8219-c64fc1c56288","Type":"ContainerStarted","Data":"c56fff334bc588a11b98225efc0287943ea487dbf7dd07e2458d57241e7b9897"}
Oct 30 21:09:07.849146 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:07.848929    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7" event={"ID":"be86287d-e81f-4b44-8219-c64fc1c56288","Type":"ContainerStarted","Data":"5aa206e0cddc7f36f8f2b4f8e7691dd44a11971cf9ead2d792761621f6fb98ab"}
Oct 30 21:09:07.950513 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:07.950479    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:09:07.967489 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:07.967433    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7" podStartSLOduration=1.967421358 podStartE2EDuration="1.967421358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:07.966861334 +0000 UTC m=+2559.883882868" watchObservedRunningTime="2026-10-30 21:09:07.967421358 +0000 UTC m=+2559.884442893"
Oct 30 21:09:08.955154 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:08.955125    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:09:09.861056 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:09.861025    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:09:09.862019 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:09.862003    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:09:10.969184 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:10.969149    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:11.187731 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:11.187695    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:11.439514 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:11.439481    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:16.728062 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:09:16.728022    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:09:17.461552 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:17.461516    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:17.667248 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:17.667213    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:17.862451 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:17.862413    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7919c9-75b57t8rdn"]
Oct 30 21:09:17.866585 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:17.866562    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7919c9-75b57t8rdn"
Oct 30 21:09:17.975330 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:17.975291    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95e32220-e262-417b-a0b8-c79e790c8600-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-7919c9-75b57t8rdn\" (UID: \"95e32220-e262-417b-a0b8-c79e790c8600\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7919c9-75b57t8rdn"
Oct 30 21:09:17.977295 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:17.975584    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8269x\" (UniqueName: \"kubernetes.io/projected/95e32220-e262-417b-a0b8-c79e790c8600-kube-api-access-8269x\") pod \"payload-pre-processing-e2e-worker-single-main-7919c9-75b57t8rdn\" (UID: \"95e32220-e262-417b-a0b8-c79e790c8600\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7919c9-75b57t8rdn"
Oct 30 21:09:17.977295 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:17.976064    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-e2d1a44f-dockercfg-7fbc8\""
Oct 30 21:09:17.977295 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:17.976143    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-7919c9\""
Oct 30 21:09:17.983120 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:17.983018    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7919c9-75b57t8rdn"]
Oct 30 21:09:18.076388 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:18.076352    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95e32220-e262-417b-a0b8-c79e790c8600-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-7919c9-75b57t8rdn\" (UID: \"95e32220-e262-417b-a0b8-c79e790c8600\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7919c9-75b57t8rdn"
Oct 30 21:09:18.076388 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:18.076387    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8269x\" (UniqueName: \"kubernetes.io/projected/95e32220-e262-417b-a0b8-c79e790c8600-kube-api-access-8269x\") pod \"payload-pre-processing-e2e-worker-single-main-7919c9-75b57t8rdn\" (UID: \"95e32220-e262-417b-a0b8-c79e790c8600\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7919c9-75b57t8rdn"
Oct 30 21:09:18.077079 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:18.077059    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95e32220-e262-417b-a0b8-c79e790c8600-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-7919c9-75b57t8rdn\" (UID: \"95e32220-e262-417b-a0b8-c79e790c8600\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7919c9-75b57t8rdn"
Oct 30 21:09:18.136863 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:18.136757    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8269x\" (UniqueName: \"kubernetes.io/projected/95e32220-e262-417b-a0b8-c79e790c8600-kube-api-access-8269x\") pod \"payload-pre-processing-e2e-worker-single-main-7919c9-75b57t8rdn\" (UID: \"95e32220-e262-417b-a0b8-c79e790c8600\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7919c9-75b57t8rdn"
Oct 30 21:09:18.180763 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:18.180736    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7919c9-75b57t8rdn"
Oct 30 21:09:18.563794 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:18.563763    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7919c9-75b57t8rdn"]
Oct 30 21:09:18.565128 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:09:18.565100    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95e32220_e262_417b_a0b8_c79e790c8600.slice/crio-9f7b869d1b60fa08402f68dc9555359088784e5b32da17ffacfd63e0a98c9a76 WatchSource:0}: Error finding container 9f7b869d1b60fa08402f68dc9555359088784e5b32da17ffacfd63e0a98c9a76: Status 404 returned error can't find the container with id 9f7b869d1b60fa08402f68dc9555359088784e5b32da17ffacfd63e0a98c9a76
Oct 30 21:09:18.913965 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:18.913862    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7919c9-75b57t8rdn" event={"ID":"95e32220-e262-417b-a0b8-c79e790c8600","Type":"ContainerStarted","Data":"53d641087dca9414ab86d1ebb305d522fe56bddae5a9a73e83da63024125069f"}
Oct 30 21:09:18.913965 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:18.913931    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7919c9-75b57t8rdn" event={"ID":"95e32220-e262-417b-a0b8-c79e790c8600","Type":"ContainerStarted","Data":"9f7b869d1b60fa08402f68dc9555359088784e5b32da17ffacfd63e0a98c9a76"}
Oct 30 21:09:18.913965 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:18.913957    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7919c9-75b57t8rdn"
Oct 30 21:09:19.035266 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:19.035211    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7919c9-75b57t8rdn" podStartSLOduration=2.035197311 podStartE2EDuration="2.035197311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:19.032491685 +0000 UTC m=+2570.949513222" watchObservedRunningTime="2026-10-30 21:09:19.035197311 +0000 UTC m=+2570.952218845"
Oct 30 21:09:22.586057 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:22.586015    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:23.035488 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:23.035395    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:25.055120 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:25.055084    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-flpwp"]
Oct 30 21:09:25.104195 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:25.104159    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-flpwp"]
Oct 30 21:09:25.646851 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:25.646798    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:09:26.200203 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:26.200160    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621" path="/var/lib/kubelet/pods/e6a4cf9c-bd67-4cc8-bbdf-e648a0dc7621/volumes"
Oct 30 21:09:27.614287 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:27.614237    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:29.922875 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:29.922846    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7919c9-75b57t8rdn"
Oct 30 21:09:32.605545 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:32.605514    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:34.114420 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:34.114379    5071 scope.go:122] "RemoveContainer" containerID="d3832bd9534764898de630ab20b3194da5306a10ecda73bb9aed983754587699"
Oct 30 21:09:35.857138 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:35.857102    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:36.026759 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:36.026726    5071 generic.go:356] "Generic (PLEG): container finished" podID="be86287d-e81f-4b44-8219-c64fc1c56288" containerID="c56fff334bc588a11b98225efc0287943ea487dbf7dd07e2458d57241e7b9897" exitCode=0
Oct 30 21:09:36.026968 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:36.026807    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7" event={"ID":"be86287d-e81f-4b44-8219-c64fc1c56288","Type":"ContainerDied","Data":"c56fff334bc588a11b98225efc0287943ea487dbf7dd07e2458d57241e7b9897"}
Oct 30 21:09:36.027226 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:36.027212    5071 scope.go:122] "RemoveContainer" containerID="c56fff334bc588a11b98225efc0287943ea487dbf7dd07e2458d57241e7b9897"
Oct 30 21:09:36.028445 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:36.028416    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:09:36.028508 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:36.028489    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:09:36.028543 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:36.028523    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:09:36.540847 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:36.538224    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:36.719801 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:36.719753    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:37.061758 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:37.061718    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kz6v7"]
Oct 30 21:09:37.062877 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:37.062848    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7" event={"ID":"be86287d-e81f-4b44-8219-c64fc1c56288","Type":"ContainerStarted","Data":"a35577bf3abf56e9b63dbe0a54b3bdf24dfeb8cbe6d0d07304efc3da046cefb0"}
Oct 30 21:09:37.110038 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:37.110008    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kz6v7"]
Oct 30 21:09:37.954857 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:37.953918    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:09:37.954857 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:37.954440    5071 patch_prober.go:29] interesting pod/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 30 21:09:37.954857 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:37.954493    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7" podUID="be86287d-e81f-4b44-8219-c64fc1c56288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 30 21:09:38.047906 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:38.044388    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:38.200473 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:38.200436    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01ae32d3-9b0b-49ab-a1bc-3146f79084d7" path="/var/lib/kubelet/pods/01ae32d3-9b0b-49ab-a1bc-3146f79084d7/volumes"
Oct 30 21:09:38.415781 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:38.415735    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:38.953223 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:38.953181    5071 patch_prober.go:29] interesting pod/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 30 21:09:38.953417 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:38.953270    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7" podUID="be86287d-e81f-4b44-8219-c64fc1c56288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 30 21:09:39.779471 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:39.779436    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:39.861170 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:39.861132    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:09:39.952930 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:39.950196    5071 patch_prober.go:29] interesting pod/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 30 21:09:39.952930 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:39.950262    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7" podUID="be86287d-e81f-4b44-8219-c64fc1c56288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 30 21:09:40.953167 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:40.953122    5071 patch_prober.go:29] interesting pod/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 30 21:09:40.953693 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:40.953186    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7" podUID="be86287d-e81f-4b44-8219-c64fc1c56288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 30 21:09:41.950534 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:41.950482    5071 patch_prober.go:29] interesting pod/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Oct 30 21:09:41.950780 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:41.950550    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7" podUID="be86287d-e81f-4b44-8219-c64fc1c56288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Oct 30 21:09:42.954267 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:42.954237    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:09:42.955264 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:42.955236    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:09:43.139546 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:43.139513    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:43.370714 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:43.370679    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:44.561385 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:44.561349    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:09:46.297814 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:46.297777    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:46.727303 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:09:46.727221    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:09:48.119053 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:48.119010    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:53.078396 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:53.078353    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:58.092132 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:09:58.092096    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:10:02.985363 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:02.985324    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:10:08.012037 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:08.008549    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:10:12.075799 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:12.075762    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:10:12.993294 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:12.993255    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:10:15.056226 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:15.056190    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shbg5"]
Oct 30 21:10:15.105742 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:15.105708    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shbg5"]
Oct 30 21:10:16.200803 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:16.200769    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c84605e5-be76-47c2-8fe3-e439135d3396" path="/var/lib/kubelet/pods/c84605e5-be76-47c2-8fe3-e439135d3396/volumes"
Oct 30 21:10:16.727314 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:10:16.727278    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:10:17.972119 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:17.972083    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:10:22.886872 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:22.886807    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:10:26.264480 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:26.264438    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f192083-pdf82"]
Oct 30 21:10:26.269109 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:26.269067    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f192083-pdf82"
Oct 30 21:10:26.323626 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:26.323590    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f192083-pdf82"]
Oct 30 21:10:26.369555 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:26.369522    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:10:26.369555 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:26.369552    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 21:10:26.371799 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:26.371783    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:10:26.371799 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:26.371793    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:10:26.400691 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:26.400662    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/974db133-15d8-49a3-acaa-23ec86daf784-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f192083-pdf82\" (UID: \"974db133-15d8-49a3-acaa-23ec86daf784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f192083-pdf82"
Oct 30 21:10:26.400869 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:26.400793    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84w9q\" (UniqueName: \"kubernetes.io/projected/974db133-15d8-49a3-acaa-23ec86daf784-kube-api-access-84w9q\") pod \"maas-api-revoke-sub-4f192083-pdf82\" (UID: \"974db133-15d8-49a3-acaa-23ec86daf784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f192083-pdf82"
Oct 30 21:10:26.400869 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:26.400857    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/974db133-15d8-49a3-acaa-23ec86daf784-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f192083-pdf82\" (UID: \"974db133-15d8-49a3-acaa-23ec86daf784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f192083-pdf82"
Oct 30 21:10:26.501996 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:26.501963    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84w9q\" (UniqueName: \"kubernetes.io/projected/974db133-15d8-49a3-acaa-23ec86daf784-kube-api-access-84w9q\") pod \"maas-api-revoke-sub-4f192083-pdf82\" (UID: \"974db133-15d8-49a3-acaa-23ec86daf784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f192083-pdf82"
Oct 30 21:10:26.502161 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:26.502009    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/974db133-15d8-49a3-acaa-23ec86daf784-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f192083-pdf82\" (UID: \"974db133-15d8-49a3-acaa-23ec86daf784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f192083-pdf82"
Oct 30 21:10:26.502161 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:26.502063    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/974db133-15d8-49a3-acaa-23ec86daf784-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f192083-pdf82\" (UID: \"974db133-15d8-49a3-acaa-23ec86daf784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f192083-pdf82"
Oct 30 21:10:26.502678 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:26.502640    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/974db133-15d8-49a3-acaa-23ec86daf784-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f192083-pdf82\" (UID: \"974db133-15d8-49a3-acaa-23ec86daf784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f192083-pdf82"
Oct 30 21:10:26.504625 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:26.504604    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/974db133-15d8-49a3-acaa-23ec86daf784-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f192083-pdf82\" (UID: \"974db133-15d8-49a3-acaa-23ec86daf784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f192083-pdf82"
Oct 30 21:10:26.561258 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:26.561232    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84w9q\" (UniqueName: \"kubernetes.io/projected/974db133-15d8-49a3-acaa-23ec86daf784-kube-api-access-84w9q\") pod \"maas-api-revoke-sub-4f192083-pdf82\" (UID: \"974db133-15d8-49a3-acaa-23ec86daf784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f192083-pdf82"
Oct 30 21:10:26.581328 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:26.581302    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f192083-pdf82"
Oct 30 21:10:26.952143 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:26.952117    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f192083-pdf82"]
Oct 30 21:10:26.953469 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:10:26.953440    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod974db133_15d8_49a3_acaa_23ec86daf784.slice/crio-b9b06ace80498e27b71988ffc015760e0be17c88e126d62175eeb329a1d9cc63 WatchSource:0}: Error finding container b9b06ace80498e27b71988ffc015760e0be17c88e126d62175eeb329a1d9cc63: Status 404 returned error can't find the container with id b9b06ace80498e27b71988ffc015760e0be17c88e126d62175eeb329a1d9cc63
Oct 30 21:10:27.397558 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:27.397521    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f192083-pdf82" event={"ID":"974db133-15d8-49a3-acaa-23ec86daf784","Type":"ContainerStarted","Data":"bdb787102e882bdf44f43fe82ced4509f7e27b27a2e8d1b93c8559d148e878d1"}
Oct 30 21:10:27.397995 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:27.397571    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f192083-pdf82" event={"ID":"974db133-15d8-49a3-acaa-23ec86daf784","Type":"ContainerStarted","Data":"b9b06ace80498e27b71988ffc015760e0be17c88e126d62175eeb329a1d9cc63"}
Oct 30 21:10:27.513763 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:27.513649    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f192083-pdf82" podStartSLOduration=1.513630943 podStartE2EDuration="1.513630943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:27.511582651 +0000 UTC m=+2639.428604201" watchObservedRunningTime="2026-10-30 21:10:27.513630943 +0000 UTC m=+2639.430652481"
Oct 30 21:10:27.876621 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:27.876560    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:10:28.405687 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:28.405658    5071 generic.go:356] "Generic (PLEG): container finished" podID="974db133-15d8-49a3-acaa-23ec86daf784" containerID="bdb787102e882bdf44f43fe82ced4509f7e27b27a2e8d1b93c8559d148e878d1" exitCode=0
Oct 30 21:10:28.406136 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:28.405741    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f192083-pdf82" event={"ID":"974db133-15d8-49a3-acaa-23ec86daf784","Type":"ContainerDied","Data":"bdb787102e882bdf44f43fe82ced4509f7e27b27a2e8d1b93c8559d148e878d1"}
Oct 30 21:10:29.559670 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:29.559648    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f192083-pdf82"
Oct 30 21:10:29.632927 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:29.632895    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/974db133-15d8-49a3-acaa-23ec86daf784-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/974db133-15d8-49a3-acaa-23ec86daf784-maas-api-openshift-ca\") pod \"974db133-15d8-49a3-acaa-23ec86daf784\" (UID: \"974db133-15d8-49a3-acaa-23ec86daf784\") "
Oct 30 21:10:29.633115 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:29.633096    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/974db133-15d8-49a3-acaa-23ec86daf784-kube-api-access-84w9q\" (UniqueName: \"kubernetes.io/projected/974db133-15d8-49a3-acaa-23ec86daf784-kube-api-access-84w9q\") pod \"974db133-15d8-49a3-acaa-23ec86daf784\" (UID: \"974db133-15d8-49a3-acaa-23ec86daf784\") "
Oct 30 21:10:29.633202 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:29.633141    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/974db133-15d8-49a3-acaa-23ec86daf784-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/974db133-15d8-49a3-acaa-23ec86daf784-maas-api-xks-ca\") pod \"974db133-15d8-49a3-acaa-23ec86daf784\" (UID: \"974db133-15d8-49a3-acaa-23ec86daf784\") "
Oct 30 21:10:29.633303 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:29.633279    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/974db133-15d8-49a3-acaa-23ec86daf784-maas-api-openshift-ca" pod "974db133-15d8-49a3-acaa-23ec86daf784" (UID: "974db133-15d8-49a3-acaa-23ec86daf784"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:10:29.633477 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:29.633461    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/974db133-15d8-49a3-acaa-23ec86daf784-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:10:29.635417 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:29.635395    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/974db133-15d8-49a3-acaa-23ec86daf784-kube-api-access-84w9q" pod "974db133-15d8-49a3-acaa-23ec86daf784" (UID: "974db133-15d8-49a3-acaa-23ec86daf784"). InnerVolumeSpecName "kube-api-access-84w9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:10:29.635417 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:29.635409    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/974db133-15d8-49a3-acaa-23ec86daf784-maas-api-xks-ca" pod "974db133-15d8-49a3-acaa-23ec86daf784" (UID: "974db133-15d8-49a3-acaa-23ec86daf784"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:10:29.734847 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:29.734735    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84w9q\" (UniqueName: \"kubernetes.io/projected/974db133-15d8-49a3-acaa-23ec86daf784-kube-api-access-84w9q\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:10:29.734847 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:29.734769    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/974db133-15d8-49a3-acaa-23ec86daf784-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:10:30.418785 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:30.418743    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f192083-pdf82"
Oct 30 21:10:30.418785 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:30.418774    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f192083-pdf82" event={"ID":"974db133-15d8-49a3-acaa-23ec86daf784","Type":"ContainerDied","Data":"b9b06ace80498e27b71988ffc015760e0be17c88e126d62175eeb329a1d9cc63"}
Oct 30 21:10:30.419010 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:30.418834    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9b06ace80498e27b71988ffc015760e0be17c88e126d62175eeb329a1d9cc63"
Oct 30 21:10:32.770571 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:32.770530    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:10:34.139694 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:34.139659    5071 scope.go:122] "RemoveContainer" containerID="a05103928f84aabe72fb6f57ecb8b2b1aa3148eace7742bffd02449050a258e2"
Oct 30 21:10:34.148994 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:34.148973    5071 scope.go:122] "RemoveContainer" containerID="0f3fa36b0d55e648ef0fb9d4d8ec347da7f10fc3b835233cc7698c2b61ee3bb1"
Oct 30 21:10:34.157497 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:34.157474    5071 scope.go:122] "RemoveContainer" containerID="d6df5ba86e128843c6993d2c39afbcbd55de74bbcd8db6a15c8e3c7724d12ec4"
Oct 30 21:10:34.165483 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:34.165463    5071 scope.go:122] "RemoveContainer" containerID="876a9d31aa718903a53e7c4d72d867e612daf5c65efeb38ab0e4d07ead9d5a8e"
Oct 30 21:10:37.711977 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:37.711893    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:10:37.996522 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:37.996436    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:10:38.515290 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:38.515258    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7919c9-65s67"]
Oct 30 21:10:38.515722 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:38.515708    5071 cpu_manager.go:414] "containerMap: removing container" podUID="974db133-15d8-49a3-acaa-23ec86daf784" containerName="revoke-keys"
Oct 30 21:10:38.515782 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:38.515724    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="974db133-15d8-49a3-acaa-23ec86daf784" containerName="revoke-keys"
Oct 30 21:10:38.515845 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:38.515813    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="974db133-15d8-49a3-acaa-23ec86daf784" containerName="revoke-keys"
Oct 30 21:10:38.518899 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:38.518882    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7919c9-65s67"
Oct 30 21:10:38.578249 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:38.578219    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7919c9-65s67"]
Oct 30 21:10:38.622396 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:38.622352    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km45c\" (UniqueName: \"kubernetes.io/projected/89a1fcca-e6ef-4b36-a0fb-1323bf776680-kube-api-access-km45c\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7919c9-65s67\" (UID: \"89a1fcca-e6ef-4b36-a0fb-1323bf776680\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7919c9-65s67"
Oct 30 21:10:38.622567 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:38.622406    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89a1fcca-e6ef-4b36-a0fb-1323bf776680-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7919c9-65s67\" (UID: \"89a1fcca-e6ef-4b36-a0fb-1323bf776680\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7919c9-65s67"
Oct 30 21:10:38.622567 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:38.622503    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89a1fcca-e6ef-4b36-a0fb-1323bf776680-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7919c9-65s67\" (UID: \"89a1fcca-e6ef-4b36-a0fb-1323bf776680\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7919c9-65s67"
Oct 30 21:10:38.624258 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:38.624229    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:10:38.624400 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:38.624378    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 21:10:38.624400 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:38.624391    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:10:38.624542 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:38.624409    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:10:38.723057 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:38.723022    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89a1fcca-e6ef-4b36-a0fb-1323bf776680-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7919c9-65s67\" (UID: \"89a1fcca-e6ef-4b36-a0fb-1323bf776680\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7919c9-65s67"
Oct 30 21:10:38.723630 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:38.723087    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89a1fcca-e6ef-4b36-a0fb-1323bf776680-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7919c9-65s67\" (UID: \"89a1fcca-e6ef-4b36-a0fb-1323bf776680\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7919c9-65s67"
Oct 30 21:10:38.723630 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:38.723180    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km45c\" (UniqueName: \"kubernetes.io/projected/89a1fcca-e6ef-4b36-a0fb-1323bf776680-kube-api-access-km45c\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7919c9-65s67\" (UID: \"89a1fcca-e6ef-4b36-a0fb-1323bf776680\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7919c9-65s67"
Oct 30 21:10:38.723630 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:38.723476    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:10:38.723922 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:38.723900    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89a1fcca-e6ef-4b36-a0fb-1323bf776680-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7919c9-65s67\" (UID: \"89a1fcca-e6ef-4b36-a0fb-1323bf776680\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7919c9-65s67"
Oct 30 21:10:38.725663 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:38.725639    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89a1fcca-e6ef-4b36-a0fb-1323bf776680-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7919c9-65s67\" (UID: \"89a1fcca-e6ef-4b36-a0fb-1323bf776680\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7919c9-65s67"
Oct 30 21:10:38.781794 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:38.781721    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km45c\" (UniqueName: \"kubernetes.io/projected/89a1fcca-e6ef-4b36-a0fb-1323bf776680-kube-api-access-km45c\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7919c9-65s67\" (UID: \"89a1fcca-e6ef-4b36-a0fb-1323bf776680\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7919c9-65s67"
Oct 30 21:10:38.830549 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:38.830525    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7919c9-65s67"
Oct 30 21:10:39.208461 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:39.208435    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7919c9-65s67"]
Oct 30 21:10:39.210535 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:10:39.210507    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89a1fcca_e6ef_4b36_a0fb_1323bf776680.slice/crio-d5bb4da16642d665d193dad705d20e155c397b9f2fbaa1452f76b466cf865bee WatchSource:0}: Error finding container d5bb4da16642d665d193dad705d20e155c397b9f2fbaa1452f76b466cf865bee: Status 404 returned error can't find the container with id d5bb4da16642d665d193dad705d20e155c397b9f2fbaa1452f76b466cf865bee
Oct 30 21:10:39.470112 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:39.470037    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7919c9-65s67" event={"ID":"89a1fcca-e6ef-4b36-a0fb-1323bf776680","Type":"ContainerStarted","Data":"d5bb4da16642d665d193dad705d20e155c397b9f2fbaa1452f76b466cf865bee"}
Oct 30 21:10:40.476786 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:40.476752    5071 generic.go:356] "Generic (PLEG): container finished" podID="89a1fcca-e6ef-4b36-a0fb-1323bf776680" containerID="2f404813e989e4c141da77dd43bd2299c4a59c607ee5be9f2a8c98b41adc1c0c" exitCode=0
Oct 30 21:10:40.477210 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:40.476833    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7919c9-65s67" event={"ID":"89a1fcca-e6ef-4b36-a0fb-1323bf776680","Type":"ContainerDied","Data":"2f404813e989e4c141da77dd43bd2299c4a59c607ee5be9f2a8c98b41adc1c0c"}
Oct 30 21:10:41.252198 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:41.252164    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:10:41.622912 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:41.622887    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7919c9-65s67"
Oct 30 21:10:41.761504 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:41.761474    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89a1fcca-e6ef-4b36-a0fb-1323bf776680-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89a1fcca-e6ef-4b36-a0fb-1323bf776680-maas-api-xks-ca\") pod \"89a1fcca-e6ef-4b36-a0fb-1323bf776680\" (UID: \"89a1fcca-e6ef-4b36-a0fb-1323bf776680\") "
Oct 30 21:10:41.761689 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:41.761521    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89a1fcca-e6ef-4b36-a0fb-1323bf776680-kube-api-access-km45c\" (UniqueName: \"kubernetes.io/projected/89a1fcca-e6ef-4b36-a0fb-1323bf776680-kube-api-access-km45c\") pod \"89a1fcca-e6ef-4b36-a0fb-1323bf776680\" (UID: \"89a1fcca-e6ef-4b36-a0fb-1323bf776680\") "
Oct 30 21:10:41.761689 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:41.761669    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89a1fcca-e6ef-4b36-a0fb-1323bf776680-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89a1fcca-e6ef-4b36-a0fb-1323bf776680-maas-api-openshift-ca\") pod \"89a1fcca-e6ef-4b36-a0fb-1323bf776680\" (UID: \"89a1fcca-e6ef-4b36-a0fb-1323bf776680\") "
Oct 30 21:10:41.762025 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:41.762000    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89a1fcca-e6ef-4b36-a0fb-1323bf776680-maas-api-openshift-ca" pod "89a1fcca-e6ef-4b36-a0fb-1323bf776680" (UID: "89a1fcca-e6ef-4b36-a0fb-1323bf776680"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:10:41.763835 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:41.763802    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89a1fcca-e6ef-4b36-a0fb-1323bf776680-maas-api-xks-ca" pod "89a1fcca-e6ef-4b36-a0fb-1323bf776680" (UID: "89a1fcca-e6ef-4b36-a0fb-1323bf776680"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:10:41.763911 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:41.763889    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89a1fcca-e6ef-4b36-a0fb-1323bf776680-kube-api-access-km45c" pod "89a1fcca-e6ef-4b36-a0fb-1323bf776680" (UID: "89a1fcca-e6ef-4b36-a0fb-1323bf776680"). InnerVolumeSpecName "kube-api-access-km45c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:10:41.862927 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:41.862893    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89a1fcca-e6ef-4b36-a0fb-1323bf776680-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:10:41.862927 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:41.862922    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89a1fcca-e6ef-4b36-a0fb-1323bf776680-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:10:41.863130 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:41.862931    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km45c\" (UniqueName: \"kubernetes.io/projected/89a1fcca-e6ef-4b36-a0fb-1323bf776680-kube-api-access-km45c\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:10:42.488907 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:42.488810    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7919c9-65s67" event={"ID":"89a1fcca-e6ef-4b36-a0fb-1323bf776680","Type":"ContainerDied","Data":"d5bb4da16642d665d193dad705d20e155c397b9f2fbaa1452f76b466cf865bee"}
Oct 30 21:10:42.488907 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:42.488887    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5bb4da16642d665d193dad705d20e155c397b9f2fbaa1452f76b466cf865bee"
Oct 30 21:10:42.488907 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:42.488838    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7919c9-65s67"
Oct 30 21:10:43.026320 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:43.026286    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:10:43.084381 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:43.084339    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6pt2s"]
Oct 30 21:10:43.084701 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:43.084682    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6pt2s"]
Oct 30 21:10:43.186084 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:43.186029    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:10:43.471160 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:43.471111    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:10:44.200946 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:44.200917    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc84820e-da50-4a2d-8cb6-6624f165d9c0" path="/var/lib/kubelet/pods/cc84820e-da50-4a2d-8cb6-6624f165d9c0/volumes"
Oct 30 21:10:46.630774 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:46.630739    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:10:46.729756 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:10:46.729713    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:10:48.070194 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:48.070155    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:10:50.189256 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:50.189215    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7919c9-75b57t8rdn"]
Oct 30 21:10:50.473121 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:50.473035    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:10:50.639588 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:50.637713    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:10:52.892566 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:52.892519    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:10:55.190275 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:55.190216    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7919c9-75b57t8rdn" podUID="95e32220-e262-417b-a0b8-c79e790c8600" containerName="payload-pre-processing" containerID="cri-o://53d641087dca9414ab86d1ebb305d522fe56bddae5a9a73e83da63024125069f" gracePeriod=25
Oct 30 21:10:57.205712 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:57.205670    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:10:58.739168 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:58.739132    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:10:59.026353 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:59.026269    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:10:59.393872 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:59.393839    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7919c9-65s67"]
Oct 30 21:10:59.443314 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:59.443279    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7919c9-65s67"]
Oct 30 21:10:59.789958 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:59.789925    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:10:59.921875 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:10:59.921804    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7919c9-75b57t8rdn" podUID="95e32220-e262-417b-a0b8-c79e790c8600" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:11:00.201111 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:00.201031    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89a1fcca-e6ef-4b36-a0fb-1323bf776680" path="/var/lib/kubelet/pods/89a1fcca-e6ef-4b36-a0fb-1323bf776680/volumes"
Oct 30 21:11:01.971123 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:01.971086    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:11:05.897624 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:05.897564    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:11:06.207455 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:06.207312    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:11:06.248242 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:06.248209    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"]
Oct 30 21:11:06.248447 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:06.248425    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7" podUID="be86287d-e81f-4b44-8219-c64fc1c56288" containerName="istio-proxy" containerID="cri-o://a35577bf3abf56e9b63dbe0a54b3bdf24dfeb8cbe6d0d07304efc3da046cefb0" gracePeriod=30
Oct 30 21:11:06.361152 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:06.361119    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:11:07.823854 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:07.823808    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g78dg"]
Oct 30 21:11:07.873487 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:07.873449    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g78dg"]
Oct 30 21:11:07.953584 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:07.953548    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q9fbn"]
Oct 30 21:11:07.954037 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:07.954023    5071 cpu_manager.go:414] "containerMap: removing container" podUID="89a1fcca-e6ef-4b36-a0fb-1323bf776680" containerName="revoke-keys"
Oct 30 21:11:07.954097 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:07.954038    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89a1fcca-e6ef-4b36-a0fb-1323bf776680" containerName="revoke-keys"
Oct 30 21:11:07.954135 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:07.954116    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="89a1fcca-e6ef-4b36-a0fb-1323bf776680" containerName="revoke-keys"
Oct 30 21:11:07.957164 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:07.957148    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q9fbn"
Oct 30 21:11:08.013062 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:08.013034    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:11:08.013239 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:08.013074    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:11:08.013239 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:08.013125    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:11:08.013509 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:08.013481    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 21:11:08.015009 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:08.014977    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q9fbn"]
Oct 30 21:11:08.054784 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:08.054757    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4smbh\" (UniqueName: \"kubernetes.io/projected/777b0a45-e119-42c9-a6ee-8fb4ed9446c9-kube-api-access-4smbh\") pod \"maas-api-revoke-sub-3d4156f1-q9fbn\" (UID: \"777b0a45-e119-42c9-a6ee-8fb4ed9446c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q9fbn"
Oct 30 21:11:08.054959 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:08.054924    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/777b0a45-e119-42c9-a6ee-8fb4ed9446c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-q9fbn\" (UID: \"777b0a45-e119-42c9-a6ee-8fb4ed9446c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q9fbn"
Oct 30 21:11:08.055097 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:08.055078    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/777b0a45-e119-42c9-a6ee-8fb4ed9446c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-q9fbn\" (UID: \"777b0a45-e119-42c9-a6ee-8fb4ed9446c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q9fbn"
Oct 30 21:11:08.156362 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:08.156267    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/777b0a45-e119-42c9-a6ee-8fb4ed9446c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-q9fbn\" (UID: \"777b0a45-e119-42c9-a6ee-8fb4ed9446c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q9fbn"
Oct 30 21:11:08.156362 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:08.156306    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4smbh\" (UniqueName: \"kubernetes.io/projected/777b0a45-e119-42c9-a6ee-8fb4ed9446c9-kube-api-access-4smbh\") pod \"maas-api-revoke-sub-3d4156f1-q9fbn\" (UID: \"777b0a45-e119-42c9-a6ee-8fb4ed9446c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q9fbn"
Oct 30 21:11:08.156622 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:08.156369    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/777b0a45-e119-42c9-a6ee-8fb4ed9446c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-q9fbn\" (UID: \"777b0a45-e119-42c9-a6ee-8fb4ed9446c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q9fbn"
Oct 30 21:11:08.157033 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:08.157006    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/777b0a45-e119-42c9-a6ee-8fb4ed9446c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-q9fbn\" (UID: \"777b0a45-e119-42c9-a6ee-8fb4ed9446c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q9fbn"
Oct 30 21:11:08.158996 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:08.158972    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/777b0a45-e119-42c9-a6ee-8fb4ed9446c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-q9fbn\" (UID: \"777b0a45-e119-42c9-a6ee-8fb4ed9446c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q9fbn"
Oct 30 21:11:08.200977 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:08.200945    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61fa5afd-c506-48bc-9166-93b0bc94cf57" path="/var/lib/kubelet/pods/61fa5afd-c506-48bc-9166-93b0bc94cf57/volumes"
Oct 30 21:11:08.214625 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:08.214596    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4smbh\" (UniqueName: \"kubernetes.io/projected/777b0a45-e119-42c9-a6ee-8fb4ed9446c9-kube-api-access-4smbh\") pod \"maas-api-revoke-sub-3d4156f1-q9fbn\" (UID: \"777b0a45-e119-42c9-a6ee-8fb4ed9446c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q9fbn"
Oct 30 21:11:08.270462 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:08.270439    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q9fbn"
Oct 30 21:11:08.649342 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:08.649318    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q9fbn"]
Oct 30 21:11:08.652941 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:11:08.652907    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod777b0a45_e119_42c9_a6ee_8fb4ed9446c9.slice/crio-5964f1db7f7f63b79fbde0aebb28aa20afb27f2e410053b25d5debf243b21edb WatchSource:0}: Error finding container 5964f1db7f7f63b79fbde0aebb28aa20afb27f2e410053b25d5debf243b21edb: Status 404 returned error can't find the container with id 5964f1db7f7f63b79fbde0aebb28aa20afb27f2e410053b25d5debf243b21edb
Oct 30 21:11:09.656349 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:09.656313    5071 generic.go:356] "Generic (PLEG): container finished" podID="777b0a45-e119-42c9-a6ee-8fb4ed9446c9" containerID="a29eb97753954fc10d388f40c67e2ea2d52650aab9d32fa953651bda337ae27b" exitCode=0
Oct 30 21:11:09.656880 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:09.656405    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q9fbn" event={"ID":"777b0a45-e119-42c9-a6ee-8fb4ed9446c9","Type":"ContainerDied","Data":"a29eb97753954fc10d388f40c67e2ea2d52650aab9d32fa953651bda337ae27b"}
Oct 30 21:11:09.656880 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:09.656497    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q9fbn" event={"ID":"777b0a45-e119-42c9-a6ee-8fb4ed9446c9","Type":"ContainerStarted","Data":"5964f1db7f7f63b79fbde0aebb28aa20afb27f2e410053b25d5debf243b21edb"}
Oct 30 21:11:09.921537 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:09.921490    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7919c9-75b57t8rdn" podUID="95e32220-e262-417b-a0b8-c79e790c8600" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:11:10.821142 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:10.821119    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q9fbn"
Oct 30 21:11:10.885022 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:10.884990    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/777b0a45-e119-42c9-a6ee-8fb4ed9446c9-kube-api-access-4smbh\" (UniqueName: \"kubernetes.io/projected/777b0a45-e119-42c9-a6ee-8fb4ed9446c9-kube-api-access-4smbh\") pod \"777b0a45-e119-42c9-a6ee-8fb4ed9446c9\" (UID: \"777b0a45-e119-42c9-a6ee-8fb4ed9446c9\") "
Oct 30 21:11:10.885022 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:10.885025    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/777b0a45-e119-42c9-a6ee-8fb4ed9446c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/777b0a45-e119-42c9-a6ee-8fb4ed9446c9-maas-api-openshift-ca\") pod \"777b0a45-e119-42c9-a6ee-8fb4ed9446c9\" (UID: \"777b0a45-e119-42c9-a6ee-8fb4ed9446c9\") "
Oct 30 21:11:10.885227 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:10.885102    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/777b0a45-e119-42c9-a6ee-8fb4ed9446c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/777b0a45-e119-42c9-a6ee-8fb4ed9446c9-maas-api-xks-ca\") pod \"777b0a45-e119-42c9-a6ee-8fb4ed9446c9\" (UID: \"777b0a45-e119-42c9-a6ee-8fb4ed9446c9\") "
Oct 30 21:11:10.885420 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:10.885397    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/777b0a45-e119-42c9-a6ee-8fb4ed9446c9-maas-api-openshift-ca" pod "777b0a45-e119-42c9-a6ee-8fb4ed9446c9" (UID: "777b0a45-e119-42c9-a6ee-8fb4ed9446c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:11:10.887413 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:10.887389    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/777b0a45-e119-42c9-a6ee-8fb4ed9446c9-kube-api-access-4smbh" pod "777b0a45-e119-42c9-a6ee-8fb4ed9446c9" (UID: "777b0a45-e119-42c9-a6ee-8fb4ed9446c9"). InnerVolumeSpecName "kube-api-access-4smbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:10.887500 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:10.887428    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/777b0a45-e119-42c9-a6ee-8fb4ed9446c9-maas-api-xks-ca" pod "777b0a45-e119-42c9-a6ee-8fb4ed9446c9" (UID: "777b0a45-e119-42c9-a6ee-8fb4ed9446c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:11:10.986396 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:10.986320    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/777b0a45-e119-42c9-a6ee-8fb4ed9446c9-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:11:10.986396 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:10.986355    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4smbh\" (UniqueName: \"kubernetes.io/projected/777b0a45-e119-42c9-a6ee-8fb4ed9446c9-kube-api-access-4smbh\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:11:10.986396 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:10.986372    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/777b0a45-e119-42c9-a6ee-8fb4ed9446c9-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:11:11.485868 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.485843    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:11:11.594449 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.594415    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be86287d-e81f-4b44-8219-c64fc1c56288-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be86287d-e81f-4b44-8219-c64fc1c56288-istio-podinfo\") pod \"be86287d-e81f-4b44-8219-c64fc1c56288\" (UID: \"be86287d-e81f-4b44-8219-c64fc1c56288\") "
Oct 30 21:11:11.594665 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.594460    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be86287d-e81f-4b44-8219-c64fc1c56288-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be86287d-e81f-4b44-8219-c64fc1c56288-istio-data\") pod \"be86287d-e81f-4b44-8219-c64fc1c56288\" (UID: \"be86287d-e81f-4b44-8219-c64fc1c56288\") "
Oct 30 21:11:11.594665 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.594498    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be86287d-e81f-4b44-8219-c64fc1c56288-istio-token\" (UniqueName: \"kubernetes.io/projected/be86287d-e81f-4b44-8219-c64fc1c56288-istio-token\") pod \"be86287d-e81f-4b44-8219-c64fc1c56288\" (UID: \"be86287d-e81f-4b44-8219-c64fc1c56288\") "
Oct 30 21:11:11.594665 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.594536    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be86287d-e81f-4b44-8219-c64fc1c56288-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be86287d-e81f-4b44-8219-c64fc1c56288-istio-envoy\") pod \"be86287d-e81f-4b44-8219-c64fc1c56288\" (UID: \"be86287d-e81f-4b44-8219-c64fc1c56288\") "
Oct 30 21:11:11.594665 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.594566    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be86287d-e81f-4b44-8219-c64fc1c56288-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be86287d-e81f-4b44-8219-c64fc1c56288-workload-socket\") pod \"be86287d-e81f-4b44-8219-c64fc1c56288\" (UID: \"be86287d-e81f-4b44-8219-c64fc1c56288\") "
Oct 30 21:11:11.594665 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.594621    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be86287d-e81f-4b44-8219-c64fc1c56288-kube-api-access-mj6kq\" (UniqueName: \"kubernetes.io/projected/be86287d-e81f-4b44-8219-c64fc1c56288-kube-api-access-mj6kq\") pod \"be86287d-e81f-4b44-8219-c64fc1c56288\" (UID: \"be86287d-e81f-4b44-8219-c64fc1c56288\") "
Oct 30 21:11:11.594665 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.594638    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be86287d-e81f-4b44-8219-c64fc1c56288-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be86287d-e81f-4b44-8219-c64fc1c56288-workload-certs\") pod \"be86287d-e81f-4b44-8219-c64fc1c56288\" (UID: \"be86287d-e81f-4b44-8219-c64fc1c56288\") "
Oct 30 21:11:11.595054 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.594701    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be86287d-e81f-4b44-8219-c64fc1c56288-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be86287d-e81f-4b44-8219-c64fc1c56288-istiod-ca-cert\") pod \"be86287d-e81f-4b44-8219-c64fc1c56288\" (UID: \"be86287d-e81f-4b44-8219-c64fc1c56288\") "
Oct 30 21:11:11.595054 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.594725    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be86287d-e81f-4b44-8219-c64fc1c56288-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be86287d-e81f-4b44-8219-c64fc1c56288-credential-socket\") pod \"be86287d-e81f-4b44-8219-c64fc1c56288\" (UID: \"be86287d-e81f-4b44-8219-c64fc1c56288\") "
Oct 30 21:11:11.595054 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.594903    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be86287d-e81f-4b44-8219-c64fc1c56288-istio-data" pod "be86287d-e81f-4b44-8219-c64fc1c56288" (UID: "be86287d-e81f-4b44-8219-c64fc1c56288"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:11:11.595227 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.595094    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be86287d-e81f-4b44-8219-c64fc1c56288-credential-socket" pod "be86287d-e81f-4b44-8219-c64fc1c56288" (UID: "be86287d-e81f-4b44-8219-c64fc1c56288"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:11:11.595227 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.595156    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be86287d-e81f-4b44-8219-c64fc1c56288-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:11:11.595512 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.595239    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be86287d-e81f-4b44-8219-c64fc1c56288-workload-socket" pod "be86287d-e81f-4b44-8219-c64fc1c56288" (UID: "be86287d-e81f-4b44-8219-c64fc1c56288"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:11:11.595512 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.595323    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be86287d-e81f-4b44-8219-c64fc1c56288-workload-certs" pod "be86287d-e81f-4b44-8219-c64fc1c56288" (UID: "be86287d-e81f-4b44-8219-c64fc1c56288"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:11:11.595512 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.595427    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be86287d-e81f-4b44-8219-c64fc1c56288-istiod-ca-cert" pod "be86287d-e81f-4b44-8219-c64fc1c56288" (UID: "be86287d-e81f-4b44-8219-c64fc1c56288"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:11:11.597063 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.597031    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be86287d-e81f-4b44-8219-c64fc1c56288-istio-envoy" pod "be86287d-e81f-4b44-8219-c64fc1c56288" (UID: "be86287d-e81f-4b44-8219-c64fc1c56288"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:11:11.597178 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.597106    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be86287d-e81f-4b44-8219-c64fc1c56288-kube-api-access-mj6kq" pod "be86287d-e81f-4b44-8219-c64fc1c56288" (UID: "be86287d-e81f-4b44-8219-c64fc1c56288"). InnerVolumeSpecName "kube-api-access-mj6kq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:11.597226 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.597199    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be86287d-e81f-4b44-8219-c64fc1c56288-istio-podinfo" pod "be86287d-e81f-4b44-8219-c64fc1c56288" (UID: "be86287d-e81f-4b44-8219-c64fc1c56288"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:11:11.597378 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.597356    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be86287d-e81f-4b44-8219-c64fc1c56288-istio-token" pod "be86287d-e81f-4b44-8219-c64fc1c56288" (UID: "be86287d-e81f-4b44-8219-c64fc1c56288"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:11.672461 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.672428    5071 generic.go:356] "Generic (PLEG): container finished" podID="be86287d-e81f-4b44-8219-c64fc1c56288" containerID="a35577bf3abf56e9b63dbe0a54b3bdf24dfeb8cbe6d0d07304efc3da046cefb0" exitCode=0
Oct 30 21:11:11.672654 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.672493    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7" event={"ID":"be86287d-e81f-4b44-8219-c64fc1c56288","Type":"ContainerDied","Data":"a35577bf3abf56e9b63dbe0a54b3bdf24dfeb8cbe6d0d07304efc3da046cefb0"}
Oct 30 21:11:11.672654 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.672566    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7" event={"ID":"be86287d-e81f-4b44-8219-c64fc1c56288","Type":"ContainerDied","Data":"5aa206e0cddc7f36f8f2b4f8e7691dd44a11971cf9ead2d792761621f6fb98ab"}
Oct 30 21:11:11.672654 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.672599    5071 scope.go:122] "RemoveContainer" containerID="a35577bf3abf56e9b63dbe0a54b3bdf24dfeb8cbe6d0d07304efc3da046cefb0"
Oct 30 21:11:11.672921 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.672674    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"
Oct 30 21:11:11.675177 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.675106    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q9fbn" event={"ID":"777b0a45-e119-42c9-a6ee-8fb4ed9446c9","Type":"ContainerDied","Data":"5964f1db7f7f63b79fbde0aebb28aa20afb27f2e410053b25d5debf243b21edb"}
Oct 30 21:11:11.675177 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.675153    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5964f1db7f7f63b79fbde0aebb28aa20afb27f2e410053b25d5debf243b21edb"
Oct 30 21:11:11.675177 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.675163    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q9fbn"
Oct 30 21:11:11.684761 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.684740    5071 scope.go:122] "RemoveContainer" containerID="c56fff334bc588a11b98225efc0287943ea487dbf7dd07e2458d57241e7b9897"
Oct 30 21:11:11.694575 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.694551    5071 scope.go:122] "RemoveContainer" containerID="a35577bf3abf56e9b63dbe0a54b3bdf24dfeb8cbe6d0d07304efc3da046cefb0"
Oct 30 21:11:11.694930 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:11:11.694903    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a35577bf3abf56e9b63dbe0a54b3bdf24dfeb8cbe6d0d07304efc3da046cefb0\": container with ID starting with a35577bf3abf56e9b63dbe0a54b3bdf24dfeb8cbe6d0d07304efc3da046cefb0 not found: ID does not exist" containerID="a35577bf3abf56e9b63dbe0a54b3bdf24dfeb8cbe6d0d07304efc3da046cefb0"
Oct 30 21:11:11.695015 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.694940    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a35577bf3abf56e9b63dbe0a54b3bdf24dfeb8cbe6d0d07304efc3da046cefb0"} err="failed to get container status \"a35577bf3abf56e9b63dbe0a54b3bdf24dfeb8cbe6d0d07304efc3da046cefb0\": rpc error: code = NotFound desc = could not find container \"a35577bf3abf56e9b63dbe0a54b3bdf24dfeb8cbe6d0d07304efc3da046cefb0\": container with ID starting with a35577bf3abf56e9b63dbe0a54b3bdf24dfeb8cbe6d0d07304efc3da046cefb0 not found: ID does not exist"
Oct 30 21:11:11.695015 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.694966    5071 scope.go:122] "RemoveContainer" containerID="c56fff334bc588a11b98225efc0287943ea487dbf7dd07e2458d57241e7b9897"
Oct 30 21:11:11.695278 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:11:11.695255    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c56fff334bc588a11b98225efc0287943ea487dbf7dd07e2458d57241e7b9897\": container with ID starting with c56fff334bc588a11b98225efc0287943ea487dbf7dd07e2458d57241e7b9897 not found: ID does not exist" containerID="c56fff334bc588a11b98225efc0287943ea487dbf7dd07e2458d57241e7b9897"
Oct 30 21:11:11.695351 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.695283    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c56fff334bc588a11b98225efc0287943ea487dbf7dd07e2458d57241e7b9897"} err="failed to get container status \"c56fff334bc588a11b98225efc0287943ea487dbf7dd07e2458d57241e7b9897\": rpc error: code = NotFound desc = could not find container \"c56fff334bc588a11b98225efc0287943ea487dbf7dd07e2458d57241e7b9897\": container with ID starting with c56fff334bc588a11b98225efc0287943ea487dbf7dd07e2458d57241e7b9897 not found: ID does not exist"
Oct 30 21:11:11.695741 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.695719    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj6kq\" (UniqueName: \"kubernetes.io/projected/be86287d-e81f-4b44-8219-c64fc1c56288-kube-api-access-mj6kq\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:11:11.695852 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.695748    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be86287d-e81f-4b44-8219-c64fc1c56288-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:11:11.695852 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.695765    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be86287d-e81f-4b44-8219-c64fc1c56288-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:11:11.695852 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.695779    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be86287d-e81f-4b44-8219-c64fc1c56288-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:11:11.695852 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.695793    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be86287d-e81f-4b44-8219-c64fc1c56288-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:11:11.695852 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.695806    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be86287d-e81f-4b44-8219-c64fc1c56288-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:11:11.695852 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.695837    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be86287d-e81f-4b44-8219-c64fc1c56288-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:11:11.695852 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.695851    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be86287d-e81f-4b44-8219-c64fc1c56288-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:11:11.840836 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.840803    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"]
Oct 30 21:11:11.847293 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:11.847232    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7919c9-openshift-default-76cddb54bfxhhw7"]
Oct 30 21:11:12.200969 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:12.200889    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be86287d-e81f-4b44-8219-c64fc1c56288" path="/var/lib/kubelet/pods/be86287d-e81f-4b44-8219-c64fc1c56288/volumes"
Oct 30 21:11:13.055657 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:13.055310    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vv5gl"]
Oct 30 21:11:13.056762 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:13.056729    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vv5gl"]
Oct 30 21:11:14.200681 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:14.200646    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9" path="/var/lib/kubelet/pods/b56a47dd-f9e7-4df1-8bc9-f3b296bb8be9/volumes"
Oct 30 21:11:15.449697 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:15.449630    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7919c9-75b57t8rdn"
Oct 30 21:11:15.535475 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:15.535446    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95e32220-e262-417b-a0b8-c79e790c8600-kube-api-access-8269x\" (UniqueName: \"kubernetes.io/projected/95e32220-e262-417b-a0b8-c79e790c8600-kube-api-access-8269x\") pod \"95e32220-e262-417b-a0b8-c79e790c8600\" (UID: \"95e32220-e262-417b-a0b8-c79e790c8600\") "
Oct 30 21:11:15.535599 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:15.535497    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95e32220-e262-417b-a0b8-c79e790c8600-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95e32220-e262-417b-a0b8-c79e790c8600-plugins-config-volume\") pod \"95e32220-e262-417b-a0b8-c79e790c8600\" (UID: \"95e32220-e262-417b-a0b8-c79e790c8600\") "
Oct 30 21:11:15.535886 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:15.535861    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95e32220-e262-417b-a0b8-c79e790c8600-plugins-config-volume" pod "95e32220-e262-417b-a0b8-c79e790c8600" (UID: "95e32220-e262-417b-a0b8-c79e790c8600"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:11:15.537629 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:15.537605    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95e32220-e262-417b-a0b8-c79e790c8600-kube-api-access-8269x" pod "95e32220-e262-417b-a0b8-c79e790c8600" (UID: "95e32220-e262-417b-a0b8-c79e790c8600"). InnerVolumeSpecName "kube-api-access-8269x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:15.636455 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:15.636419    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8269x\" (UniqueName: \"kubernetes.io/projected/95e32220-e262-417b-a0b8-c79e790c8600-kube-api-access-8269x\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:11:15.636455 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:15.636457    5071 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95e32220-e262-417b-a0b8-c79e790c8600-plugins-config-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:11:15.702163 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:15.702062    5071 generic.go:356] "Generic (PLEG): container finished" podID="95e32220-e262-417b-a0b8-c79e790c8600" containerID="53d641087dca9414ab86d1ebb305d522fe56bddae5a9a73e83da63024125069f" exitCode=0
Oct 30 21:11:15.702163 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:15.702122    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7919c9-75b57t8rdn"
Oct 30 21:11:15.702163 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:15.702156    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7919c9-75b57t8rdn" event={"ID":"95e32220-e262-417b-a0b8-c79e790c8600","Type":"ContainerDied","Data":"53d641087dca9414ab86d1ebb305d522fe56bddae5a9a73e83da63024125069f"}
Oct 30 21:11:15.702483 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:15.702200    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7919c9-75b57t8rdn" event={"ID":"95e32220-e262-417b-a0b8-c79e790c8600","Type":"ContainerDied","Data":"9f7b869d1b60fa08402f68dc9555359088784e5b32da17ffacfd63e0a98c9a76"}
Oct 30 21:11:15.702483 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:15.702229    5071 scope.go:122] "RemoveContainer" containerID="53d641087dca9414ab86d1ebb305d522fe56bddae5a9a73e83da63024125069f"
Oct 30 21:11:15.712631 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:15.712610    5071 scope.go:122] "RemoveContainer" containerID="53d641087dca9414ab86d1ebb305d522fe56bddae5a9a73e83da63024125069f"
Oct 30 21:11:15.712960 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:11:15.712939    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53d641087dca9414ab86d1ebb305d522fe56bddae5a9a73e83da63024125069f\": container with ID starting with 53d641087dca9414ab86d1ebb305d522fe56bddae5a9a73e83da63024125069f not found: ID does not exist" containerID="53d641087dca9414ab86d1ebb305d522fe56bddae5a9a73e83da63024125069f"
Oct 30 21:11:15.713020 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:15.712969    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53d641087dca9414ab86d1ebb305d522fe56bddae5a9a73e83da63024125069f"} err="failed to get container status \"53d641087dca9414ab86d1ebb305d522fe56bddae5a9a73e83da63024125069f\": rpc error: code = NotFound desc = could not find container \"53d641087dca9414ab86d1ebb305d522fe56bddae5a9a73e83da63024125069f\": container with ID starting with 53d641087dca9414ab86d1ebb305d522fe56bddae5a9a73e83da63024125069f not found: ID does not exist"
Oct 30 21:11:15.868791 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:15.868753    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7919c9-75b57t8rdn"]
Oct 30 21:11:15.874282 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:15.874253    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7919c9-75b57t8rdn"]
Oct 30 21:11:16.200525 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:16.200484    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95e32220-e262-417b-a0b8-c79e790c8600" path="/var/lib/kubelet/pods/95e32220-e262-417b-a0b8-c79e790c8600/volumes"
Oct 30 21:11:16.727618 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:11:16.727582    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:11:20.026717 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:20.026676    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:11:20.622682 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:20.622647    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:11:26.663729 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:26.663692    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:11:26.775418 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:26.775357    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9zb25"]
Oct 30 21:11:26.775627 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:26.775545    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9zb25" podUID="934f75e2-5d3c-4df4-88ea-57007d36d328" containerName="manager" containerID="cri-o://86b0d7361c1fef6a086447a5a86d69241b8b5258df80034087865320adc0812c" gracePeriod=10
Oct 30 21:11:27.047148 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:27.047121    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9zb25"
Oct 30 21:11:27.052307 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:27.052285    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/934f75e2-5d3c-4df4-88ea-57007d36d328-kube-api-access-k6mg9\" (UniqueName: \"kubernetes.io/projected/934f75e2-5d3c-4df4-88ea-57007d36d328-kube-api-access-k6mg9\") pod \"934f75e2-5d3c-4df4-88ea-57007d36d328\" (UID: \"934f75e2-5d3c-4df4-88ea-57007d36d328\") "
Oct 30 21:11:27.054429 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:27.054407    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/934f75e2-5d3c-4df4-88ea-57007d36d328-kube-api-access-k6mg9" pod "934f75e2-5d3c-4df4-88ea-57007d36d328" (UID: "934f75e2-5d3c-4df4-88ea-57007d36d328"). InnerVolumeSpecName "kube-api-access-k6mg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:27.155929 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:27.153117    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/934f75e2-5d3c-4df4-88ea-57007d36d328-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/934f75e2-5d3c-4df4-88ea-57007d36d328-extensions-socket-volume\") pod \"934f75e2-5d3c-4df4-88ea-57007d36d328\" (UID: \"934f75e2-5d3c-4df4-88ea-57007d36d328\") "
Oct 30 21:11:27.155929 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:27.153944    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/934f75e2-5d3c-4df4-88ea-57007d36d328-extensions-socket-volume" pod "934f75e2-5d3c-4df4-88ea-57007d36d328" (UID: "934f75e2-5d3c-4df4-88ea-57007d36d328"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:11:27.155929 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:27.154100    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6mg9\" (UniqueName: \"kubernetes.io/projected/934f75e2-5d3c-4df4-88ea-57007d36d328-kube-api-access-k6mg9\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:11:27.254905 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:27.254790    5071 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/934f75e2-5d3c-4df4-88ea-57007d36d328-extensions-socket-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:11:27.771712 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:27.771672    5071 generic.go:356] "Generic (PLEG): container finished" podID="934f75e2-5d3c-4df4-88ea-57007d36d328" containerID="86b0d7361c1fef6a086447a5a86d69241b8b5258df80034087865320adc0812c" exitCode=0
Oct 30 21:11:27.772224 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:27.771724    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9zb25" event={"ID":"934f75e2-5d3c-4df4-88ea-57007d36d328","Type":"ContainerDied","Data":"86b0d7361c1fef6a086447a5a86d69241b8b5258df80034087865320adc0812c"}
Oct 30 21:11:27.772224 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:27.771735    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9zb25"
Oct 30 21:11:27.772224 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:27.771766    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9zb25" event={"ID":"934f75e2-5d3c-4df4-88ea-57007d36d328","Type":"ContainerDied","Data":"77dc715e683a5ff4897c8694fbd81244b69d32af8f99b5db5c1f49c99879ffc4"}
Oct 30 21:11:27.772224 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:27.771786    5071 scope.go:122] "RemoveContainer" containerID="86b0d7361c1fef6a086447a5a86d69241b8b5258df80034087865320adc0812c"
Oct 30 21:11:27.785230 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:27.784957    5071 scope.go:122] "RemoveContainer" containerID="86b0d7361c1fef6a086447a5a86d69241b8b5258df80034087865320adc0812c"
Oct 30 21:11:27.785292 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:11:27.785258    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86b0d7361c1fef6a086447a5a86d69241b8b5258df80034087865320adc0812c\": container with ID starting with 86b0d7361c1fef6a086447a5a86d69241b8b5258df80034087865320adc0812c not found: ID does not exist" containerID="86b0d7361c1fef6a086447a5a86d69241b8b5258df80034087865320adc0812c"
Oct 30 21:11:27.785331 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:27.785286    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86b0d7361c1fef6a086447a5a86d69241b8b5258df80034087865320adc0812c"} err="failed to get container status \"86b0d7361c1fef6a086447a5a86d69241b8b5258df80034087865320adc0812c\": rpc error: code = NotFound desc = could not find container \"86b0d7361c1fef6a086447a5a86d69241b8b5258df80034087865320adc0812c\": container with ID starting with 86b0d7361c1fef6a086447a5a86d69241b8b5258df80034087865320adc0812c not found: ID does not exist"
Oct 30 21:11:27.938707 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:27.938675    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9zb25"]
Oct 30 21:11:27.988908 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:27.988875    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9zb25"]
Oct 30 21:11:28.200508 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:28.200479    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="934f75e2-5d3c-4df4-88ea-57007d36d328" path="/var/lib/kubelet/pods/934f75e2-5d3c-4df4-88ea-57007d36d328/volumes"
Oct 30 21:11:28.840833 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:28.840799    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8qvb_91fe9e06-0c30-4000-bddb-2ba89cd048cc/console-operator/2.log"
Oct 30 21:11:28.848800 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:28.848775    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q4f4h_80cae79e-b60f-4efe-bb58-e77753a457e5/ovn-acl-logging/0.log"
Oct 30 21:11:28.860853 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:28.860812    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8qvb_91fe9e06-0c30-4000-bddb-2ba89cd048cc/console-operator/2.log"
Oct 30 21:11:28.868222 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:28.868205    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q4f4h_80cae79e-b60f-4efe-bb58-e77753a457e5/ovn-acl-logging/0.log"
Oct 30 21:11:34.198569 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:34.198536    5071 scope.go:122] "RemoveContainer" containerID="b64e75088afaa915b7804b10ab2488a7ed6080182e8754aea538c46f48885634"
Oct 30 21:11:37.051340 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:37.051299    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j2w5c"]
Oct 30 21:11:37.100848 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:37.100798    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-j2w5c"]
Oct 30 21:11:38.200718 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:38.200681    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d70f6a39-1cde-4ac6-a351-79be7fe04a8c" path="/var/lib/kubelet/pods/d70f6a39-1cde-4ac6-a351-79be7fe04a8c/volumes"
Oct 30 21:11:41.175266 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:41.175229    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:11:43.772119 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:43.772078    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:11:46.726988 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:11:46.726951    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:11:49.072443 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:11:49.072396    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:12:16.728273 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:12:16.728185    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:12:30.369508 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:30.369477    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwcph"]
Oct 30 21:12:30.369997 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:30.369911    5071 cpu_manager.go:414] "containerMap: removing container" podUID="777b0a45-e119-42c9-a6ee-8fb4ed9446c9" containerName="revoke-keys"
Oct 30 21:12:30.369997 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:30.369923    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="777b0a45-e119-42c9-a6ee-8fb4ed9446c9" containerName="revoke-keys"
Oct 30 21:12:30.369997 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:30.369935    5071 cpu_manager.go:414] "containerMap: removing container" podUID="934f75e2-5d3c-4df4-88ea-57007d36d328" containerName="manager"
Oct 30 21:12:30.369997 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:30.369940    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="934f75e2-5d3c-4df4-88ea-57007d36d328" containerName="manager"
Oct 30 21:12:30.369997 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:30.369953    5071 cpu_manager.go:414] "containerMap: removing container" podUID="be86287d-e81f-4b44-8219-c64fc1c56288" containerName="istio-proxy"
Oct 30 21:12:30.369997 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:30.369958    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be86287d-e81f-4b44-8219-c64fc1c56288" containerName="istio-proxy"
Oct 30 21:12:30.369997 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:30.369971    5071 cpu_manager.go:414] "containerMap: removing container" podUID="be86287d-e81f-4b44-8219-c64fc1c56288" containerName="istio-proxy"
Oct 30 21:12:30.369997 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:30.369976    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be86287d-e81f-4b44-8219-c64fc1c56288" containerName="istio-proxy"
Oct 30 21:12:30.369997 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:30.369984    5071 cpu_manager.go:414] "containerMap: removing container" podUID="95e32220-e262-417b-a0b8-c79e790c8600" containerName="payload-pre-processing"
Oct 30 21:12:30.369997 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:30.369988    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95e32220-e262-417b-a0b8-c79e790c8600" containerName="payload-pre-processing"
Oct 30 21:12:30.370325 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:30.370045    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="777b0a45-e119-42c9-a6ee-8fb4ed9446c9" containerName="revoke-keys"
Oct 30 21:12:30.370325 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:30.370056    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="934f75e2-5d3c-4df4-88ea-57007d36d328" containerName="manager"
Oct 30 21:12:30.370325 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:30.370064    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="be86287d-e81f-4b44-8219-c64fc1c56288" containerName="istio-proxy"
Oct 30 21:12:30.370325 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:30.370071    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="be86287d-e81f-4b44-8219-c64fc1c56288" containerName="istio-proxy"
Oct 30 21:12:30.370325 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:30.370077    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="95e32220-e262-417b-a0b8-c79e790c8600" containerName="payload-pre-processing"
Oct 30 21:12:30.372950 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:30.372934    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwcph"
Oct 30 21:12:30.428175 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:30.428146    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwcph"]
Oct 30 21:12:30.460326 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:30.460289    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkv98\" (UniqueName: \"kubernetes.io/projected/ab471889-9dee-461d-a125-7cd405093e0a-kube-api-access-rkv98\") pod \"kuadrant-operator-controller-manager-678d87766f-lwcph\" (UID: \"ab471889-9dee-461d-a125-7cd405093e0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwcph"
Oct 30 21:12:30.460470 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:30.460348    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab471889-9dee-461d-a125-7cd405093e0a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-lwcph\" (UID: \"ab471889-9dee-461d-a125-7cd405093e0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwcph"
Oct 30 21:12:30.474725 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:30.474697    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-f5v2p\""
Oct 30 21:12:30.561164 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:30.561128    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkv98\" (UniqueName: \"kubernetes.io/projected/ab471889-9dee-461d-a125-7cd405093e0a-kube-api-access-rkv98\") pod \"kuadrant-operator-controller-manager-678d87766f-lwcph\" (UID: \"ab471889-9dee-461d-a125-7cd405093e0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwcph"
Oct 30 21:12:30.561339 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:30.561183    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab471889-9dee-461d-a125-7cd405093e0a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-lwcph\" (UID: \"ab471889-9dee-461d-a125-7cd405093e0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwcph"
Oct 30 21:12:30.561580 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:30.561558    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab471889-9dee-461d-a125-7cd405093e0a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-lwcph\" (UID: \"ab471889-9dee-461d-a125-7cd405093e0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwcph"
Oct 30 21:12:30.619705 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:30.619633    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkv98\" (UniqueName: \"kubernetes.io/projected/ab471889-9dee-461d-a125-7cd405093e0a-kube-api-access-rkv98\") pod \"kuadrant-operator-controller-manager-678d87766f-lwcph\" (UID: \"ab471889-9dee-461d-a125-7cd405093e0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwcph"
Oct 30 21:12:30.685911 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:30.685881    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwcph"
Oct 30 21:12:31.164618 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:31.164588    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwcph"]
Oct 30 21:12:31.166621 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:12:31.166591    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab471889_9dee_461d_a125_7cd405093e0a.slice/crio-0000b8cdf889251d3f26ecab69782c32abb176326cc75022d7c05c491213743d WatchSource:0}: Error finding container 0000b8cdf889251d3f26ecab69782c32abb176326cc75022d7c05c491213743d: Status 404 returned error can't find the container with id 0000b8cdf889251d3f26ecab69782c32abb176326cc75022d7c05c491213743d
Oct 30 21:12:32.117086 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:32.117042    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwcph" event={"ID":"ab471889-9dee-461d-a125-7cd405093e0a","Type":"ContainerStarted","Data":"76120fa01a2dd7033b74c0a905da0d9133628c550eae0402c86c168fafec9736"}
Oct 30 21:12:32.117543 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:32.117108    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwcph" event={"ID":"ab471889-9dee-461d-a125-7cd405093e0a","Type":"ContainerStarted","Data":"0000b8cdf889251d3f26ecab69782c32abb176326cc75022d7c05c491213743d"}
Oct 30 21:12:32.117543 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:32.117138    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwcph"
Oct 30 21:12:32.334118 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:32.334043    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwcph" podStartSLOduration=2.334028138 podStartE2EDuration="2.334028138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:32.283891203 +0000 UTC m=+2764.200912736" watchObservedRunningTime="2026-10-30 21:12:32.334028138 +0000 UTC m=+2764.251049674"
Oct 30 21:12:34.232734 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:34.232703    5071 scope.go:122] "RemoveContainer" containerID="1b54e5461f3ba1d9e7d59af536c5a8b51b4d7bbd4dca3e7159314703d784d4c7"
Oct 30 21:12:34.242113 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:34.242092    5071 scope.go:122] "RemoveContainer" containerID="eea31acbd6e4e02956845f7360b736252e02df40335b72d781752c80bf3ff7a6"
Oct 30 21:12:41.555562 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:41.555518    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:12:43.126026 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:43.125991    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lwcph"
Oct 30 21:12:44.148505 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:44.148472    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:12:46.727288 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:12:46.727252    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:12:49.417045 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:49.417007    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:12:53.450334 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:53.450301    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:12:57.757748 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:57.757710    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cswg"]
Oct 30 21:12:57.761254 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:57.761236    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cswg"
Oct 30 21:12:57.801651 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:57.801625    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/297b7ce4-811b-4092-bd6b-2fb22b9f7cea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-6cswg\" (UID: \"297b7ce4-811b-4092-bd6b-2fb22b9f7cea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cswg"
Oct 30 21:12:57.801851 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:57.801673    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4bd5\" (UniqueName: \"kubernetes.io/projected/297b7ce4-811b-4092-bd6b-2fb22b9f7cea-kube-api-access-v4bd5\") pod \"maas-api-revoke-sub-d576e68b-6cswg\" (UID: \"297b7ce4-811b-4092-bd6b-2fb22b9f7cea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cswg"
Oct 30 21:12:57.801939 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:57.801850    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/297b7ce4-811b-4092-bd6b-2fb22b9f7cea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-6cswg\" (UID: \"297b7ce4-811b-4092-bd6b-2fb22b9f7cea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cswg"
Oct 30 21:12:57.820698 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:57.820661    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cswg"]
Oct 30 21:12:57.866167 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:57.866136    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:12:57.866167 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:57.866159    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:12:57.866401 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:57.866249    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 21:12:57.866401 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:57.866317    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:12:57.902868 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:57.902836    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4bd5\" (UniqueName: \"kubernetes.io/projected/297b7ce4-811b-4092-bd6b-2fb22b9f7cea-kube-api-access-v4bd5\") pod \"maas-api-revoke-sub-d576e68b-6cswg\" (UID: \"297b7ce4-811b-4092-bd6b-2fb22b9f7cea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cswg"
Oct 30 21:12:57.903045 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:57.902905    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/297b7ce4-811b-4092-bd6b-2fb22b9f7cea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-6cswg\" (UID: \"297b7ce4-811b-4092-bd6b-2fb22b9f7cea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cswg"
Oct 30 21:12:57.903045 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:57.902957    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/297b7ce4-811b-4092-bd6b-2fb22b9f7cea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-6cswg\" (UID: \"297b7ce4-811b-4092-bd6b-2fb22b9f7cea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cswg"
Oct 30 21:12:57.903610 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:57.903585    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/297b7ce4-811b-4092-bd6b-2fb22b9f7cea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-6cswg\" (UID: \"297b7ce4-811b-4092-bd6b-2fb22b9f7cea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cswg"
Oct 30 21:12:57.905536 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:57.905518    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/297b7ce4-811b-4092-bd6b-2fb22b9f7cea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-6cswg\" (UID: \"297b7ce4-811b-4092-bd6b-2fb22b9f7cea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cswg"
Oct 30 21:12:57.961553 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:57.961526    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4bd5\" (UniqueName: \"kubernetes.io/projected/297b7ce4-811b-4092-bd6b-2fb22b9f7cea-kube-api-access-v4bd5\") pod \"maas-api-revoke-sub-d576e68b-6cswg\" (UID: \"297b7ce4-811b-4092-bd6b-2fb22b9f7cea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cswg"
Oct 30 21:12:58.074589 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:58.074564    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cswg"
Oct 30 21:12:58.446959 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:58.446932    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cswg"]
Oct 30 21:12:58.448667 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:12:58.448639    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod297b7ce4_811b_4092_bd6b_2fb22b9f7cea.slice/crio-8c88713718f3625aa6d81449ae2cbcaa470822b7335635f0b135a5c7d946f740 WatchSource:0}: Error finding container 8c88713718f3625aa6d81449ae2cbcaa470822b7335635f0b135a5c7d946f740: Status 404 returned error can't find the container with id 8c88713718f3625aa6d81449ae2cbcaa470822b7335635f0b135a5c7d946f740
Oct 30 21:12:59.283803 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:59.283769    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cswg" event={"ID":"297b7ce4-811b-4092-bd6b-2fb22b9f7cea","Type":"ContainerStarted","Data":"c277f9da25e9d03ce37810ec6db35950a98d68af1027845b785a8f39164036ab"}
Oct 30 21:12:59.284231 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:59.283838    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cswg" event={"ID":"297b7ce4-811b-4092-bd6b-2fb22b9f7cea","Type":"ContainerStarted","Data":"8c88713718f3625aa6d81449ae2cbcaa470822b7335635f0b135a5c7d946f740"}
Oct 30 21:12:59.399454 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:12:59.399392    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cswg" podStartSLOduration=2.39937415 podStartE2EDuration="2.39937415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:59.39744782 +0000 UTC m=+2791.314469355" watchObservedRunningTime="2026-10-30 21:12:59.39937415 +0000 UTC m=+2791.316395688"
Oct 30 21:13:00.291433 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:00.291396    5071 generic.go:356] "Generic (PLEG): container finished" podID="297b7ce4-811b-4092-bd6b-2fb22b9f7cea" containerID="c277f9da25e9d03ce37810ec6db35950a98d68af1027845b785a8f39164036ab" exitCode=0
Oct 30 21:13:00.291903 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:00.291482    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cswg" event={"ID":"297b7ce4-811b-4092-bd6b-2fb22b9f7cea","Type":"ContainerDied","Data":"c277f9da25e9d03ce37810ec6db35950a98d68af1027845b785a8f39164036ab"}
Oct 30 21:13:01.438474 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:01.438445    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cswg"
Oct 30 21:13:01.538711 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:01.538671    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/297b7ce4-811b-4092-bd6b-2fb22b9f7cea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/297b7ce4-811b-4092-bd6b-2fb22b9f7cea-maas-api-xks-ca\") pod \"297b7ce4-811b-4092-bd6b-2fb22b9f7cea\" (UID: \"297b7ce4-811b-4092-bd6b-2fb22b9f7cea\") "
Oct 30 21:13:01.538953 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:01.538908    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/297b7ce4-811b-4092-bd6b-2fb22b9f7cea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/297b7ce4-811b-4092-bd6b-2fb22b9f7cea-maas-api-openshift-ca\") pod \"297b7ce4-811b-4092-bd6b-2fb22b9f7cea\" (UID: \"297b7ce4-811b-4092-bd6b-2fb22b9f7cea\") "
Oct 30 21:13:01.539035 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:01.538972    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/297b7ce4-811b-4092-bd6b-2fb22b9f7cea-kube-api-access-v4bd5\" (UniqueName: \"kubernetes.io/projected/297b7ce4-811b-4092-bd6b-2fb22b9f7cea-kube-api-access-v4bd5\") pod \"297b7ce4-811b-4092-bd6b-2fb22b9f7cea\" (UID: \"297b7ce4-811b-4092-bd6b-2fb22b9f7cea\") "
Oct 30 21:13:01.539307 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:01.539279    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/297b7ce4-811b-4092-bd6b-2fb22b9f7cea-maas-api-openshift-ca" pod "297b7ce4-811b-4092-bd6b-2fb22b9f7cea" (UID: "297b7ce4-811b-4092-bd6b-2fb22b9f7cea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:13:01.541042 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:01.541017    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/297b7ce4-811b-4092-bd6b-2fb22b9f7cea-maas-api-xks-ca" pod "297b7ce4-811b-4092-bd6b-2fb22b9f7cea" (UID: "297b7ce4-811b-4092-bd6b-2fb22b9f7cea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:13:01.541267 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:01.541248    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/297b7ce4-811b-4092-bd6b-2fb22b9f7cea-kube-api-access-v4bd5" pod "297b7ce4-811b-4092-bd6b-2fb22b9f7cea" (UID: "297b7ce4-811b-4092-bd6b-2fb22b9f7cea"). InnerVolumeSpecName "kube-api-access-v4bd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:01.639970 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:01.639926    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/297b7ce4-811b-4092-bd6b-2fb22b9f7cea-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:13:01.639970 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:01.639962    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4bd5\" (UniqueName: \"kubernetes.io/projected/297b7ce4-811b-4092-bd6b-2fb22b9f7cea-kube-api-access-v4bd5\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:13:01.639970 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:01.639974    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/297b7ce4-811b-4092-bd6b-2fb22b9f7cea-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:13:02.303492 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:02.303459    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cswg"
Oct 30 21:13:02.303680 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:02.303490    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cswg" event={"ID":"297b7ce4-811b-4092-bd6b-2fb22b9f7cea","Type":"ContainerDied","Data":"8c88713718f3625aa6d81449ae2cbcaa470822b7335635f0b135a5c7d946f740"}
Oct 30 21:13:02.303680 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:02.303545    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c88713718f3625aa6d81449ae2cbcaa470822b7335635f0b135a5c7d946f740"
Oct 30 21:13:10.051252 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:10.051216    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pdgs9"]
Oct 30 21:13:10.100609 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:10.100577    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pdgs9"]
Oct 30 21:13:10.200227 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:10.200199    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c" path="/var/lib/kubelet/pods/7a9e0b21-0a11-4046-ba1e-2a5631c5ae8c/volumes"
Oct 30 21:13:10.550960 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:10.550931    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:13:16.810847 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:13:16.810706    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:13:18.683891 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:18.683852    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:13:22.050970 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:22.050930    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v9xhv"]
Oct 30 21:13:22.100745 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:22.100712    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-v9xhv"]
Oct 30 21:13:22.201936 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:22.201905    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c764024-4a55-4350-92df-03d5bf116fe7" path="/var/lib/kubelet/pods/2c764024-4a55-4350-92df-03d5bf116fe7/volumes"
Oct 30 21:13:26.694019 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:26.693979    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmh6k"]
Oct 30 21:13:26.694551 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:26.694422    5071 cpu_manager.go:414] "containerMap: removing container" podUID="297b7ce4-811b-4092-bd6b-2fb22b9f7cea" containerName="revoke-keys"
Oct 30 21:13:26.694551 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:26.694434    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="297b7ce4-811b-4092-bd6b-2fb22b9f7cea" containerName="revoke-keys"
Oct 30 21:13:26.694551 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:26.694509    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="297b7ce4-811b-4092-bd6b-2fb22b9f7cea" containerName="revoke-keys"
Oct 30 21:13:26.697735 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:26.697718    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmh6k"
Oct 30 21:13:26.755362 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:26.755326    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmh6k"]
Oct 30 21:13:26.802100 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:26.802069    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:13:26.802100 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:26.802071    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 21:13:26.802100 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:26.802097    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:13:26.802345 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:26.802069    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:13:26.838437 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:26.838405    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbbadd11-9ab2-4424-a66a-82f3d29423f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-hmh6k\" (UID: \"bbbadd11-9ab2-4424-a66a-82f3d29423f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmh6k"
Oct 30 21:13:26.838608 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:26.838448    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9kqh\" (UniqueName: \"kubernetes.io/projected/bbbadd11-9ab2-4424-a66a-82f3d29423f9-kube-api-access-b9kqh\") pod \"maas-api-revoke-sub-d2bc93bc-hmh6k\" (UID: \"bbbadd11-9ab2-4424-a66a-82f3d29423f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmh6k"
Oct 30 21:13:26.838608 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:26.838473    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbbadd11-9ab2-4424-a66a-82f3d29423f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-hmh6k\" (UID: \"bbbadd11-9ab2-4424-a66a-82f3d29423f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmh6k"
Oct 30 21:13:26.939751 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:26.939716    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9kqh\" (UniqueName: \"kubernetes.io/projected/bbbadd11-9ab2-4424-a66a-82f3d29423f9-kube-api-access-b9kqh\") pod \"maas-api-revoke-sub-d2bc93bc-hmh6k\" (UID: \"bbbadd11-9ab2-4424-a66a-82f3d29423f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmh6k"
Oct 30 21:13:26.939960 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:26.939760    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbbadd11-9ab2-4424-a66a-82f3d29423f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-hmh6k\" (UID: \"bbbadd11-9ab2-4424-a66a-82f3d29423f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmh6k"
Oct 30 21:13:26.939960 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:26.939876    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbbadd11-9ab2-4424-a66a-82f3d29423f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-hmh6k\" (UID: \"bbbadd11-9ab2-4424-a66a-82f3d29423f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmh6k"
Oct 30 21:13:26.940528 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:26.940508    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbbadd11-9ab2-4424-a66a-82f3d29423f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-hmh6k\" (UID: \"bbbadd11-9ab2-4424-a66a-82f3d29423f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmh6k"
Oct 30 21:13:26.942294 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:26.942270    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbbadd11-9ab2-4424-a66a-82f3d29423f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-hmh6k\" (UID: \"bbbadd11-9ab2-4424-a66a-82f3d29423f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmh6k"
Oct 30 21:13:26.998057 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:26.997997    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9kqh\" (UniqueName: \"kubernetes.io/projected/bbbadd11-9ab2-4424-a66a-82f3d29423f9-kube-api-access-b9kqh\") pod \"maas-api-revoke-sub-d2bc93bc-hmh6k\" (UID: \"bbbadd11-9ab2-4424-a66a-82f3d29423f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmh6k"
Oct 30 21:13:27.010883 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:27.010864    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmh6k"
Oct 30 21:13:27.381664 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:27.381635    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmh6k"]
Oct 30 21:13:27.383291 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:13:27.383267    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbbadd11_9ab2_4424_a66a_82f3d29423f9.slice/crio-42c0f7196fa45bff577380132b8ed51834f70db21f534f5e7b4fd962508ab9e5 WatchSource:0}: Error finding container 42c0f7196fa45bff577380132b8ed51834f70db21f534f5e7b4fd962508ab9e5: Status 404 returned error can't find the container with id 42c0f7196fa45bff577380132b8ed51834f70db21f534f5e7b4fd962508ab9e5
Oct 30 21:13:27.472237 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:27.472200    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmh6k" event={"ID":"bbbadd11-9ab2-4424-a66a-82f3d29423f9","Type":"ContainerStarted","Data":"42c0f7196fa45bff577380132b8ed51834f70db21f534f5e7b4fd962508ab9e5"}
Oct 30 21:13:28.479142 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:28.479054    5071 generic.go:356] "Generic (PLEG): container finished" podID="bbbadd11-9ab2-4424-a66a-82f3d29423f9" containerID="ea848bed7c29d88e949ec28cf6d4461115bd0beb09ffe1ff2fa40709fb1e02b5" exitCode=0
Oct 30 21:13:28.479518 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:28.479136    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmh6k" event={"ID":"bbbadd11-9ab2-4424-a66a-82f3d29423f9","Type":"ContainerDied","Data":"ea848bed7c29d88e949ec28cf6d4461115bd0beb09ffe1ff2fa40709fb1e02b5"}
Oct 30 21:13:29.617981 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:29.617955    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmh6k"
Oct 30 21:13:29.766888 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:29.766795    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbbadd11-9ab2-4424-a66a-82f3d29423f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbbadd11-9ab2-4424-a66a-82f3d29423f9-maas-api-xks-ca\") pod \"bbbadd11-9ab2-4424-a66a-82f3d29423f9\" (UID: \"bbbadd11-9ab2-4424-a66a-82f3d29423f9\") "
Oct 30 21:13:29.767067 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:29.766935    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbbadd11-9ab2-4424-a66a-82f3d29423f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbbadd11-9ab2-4424-a66a-82f3d29423f9-maas-api-openshift-ca\") pod \"bbbadd11-9ab2-4424-a66a-82f3d29423f9\" (UID: \"bbbadd11-9ab2-4424-a66a-82f3d29423f9\") "
Oct 30 21:13:29.767067 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:29.766993    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbbadd11-9ab2-4424-a66a-82f3d29423f9-kube-api-access-b9kqh\" (UniqueName: \"kubernetes.io/projected/bbbadd11-9ab2-4424-a66a-82f3d29423f9-kube-api-access-b9kqh\") pod \"bbbadd11-9ab2-4424-a66a-82f3d29423f9\" (UID: \"bbbadd11-9ab2-4424-a66a-82f3d29423f9\") "
Oct 30 21:13:29.767384 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:29.767354    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbbadd11-9ab2-4424-a66a-82f3d29423f9-maas-api-openshift-ca" pod "bbbadd11-9ab2-4424-a66a-82f3d29423f9" (UID: "bbbadd11-9ab2-4424-a66a-82f3d29423f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:13:29.769119 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:29.769096    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbbadd11-9ab2-4424-a66a-82f3d29423f9-maas-api-xks-ca" pod "bbbadd11-9ab2-4424-a66a-82f3d29423f9" (UID: "bbbadd11-9ab2-4424-a66a-82f3d29423f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:13:29.769229 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:29.769206    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbadd11-9ab2-4424-a66a-82f3d29423f9-kube-api-access-b9kqh" pod "bbbadd11-9ab2-4424-a66a-82f3d29423f9" (UID: "bbbadd11-9ab2-4424-a66a-82f3d29423f9"). InnerVolumeSpecName "kube-api-access-b9kqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:29.868087 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:29.868043    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbbadd11-9ab2-4424-a66a-82f3d29423f9-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:13:29.868087 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:29.868079    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbbadd11-9ab2-4424-a66a-82f3d29423f9-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:13:29.868087 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:29.868093    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9kqh\" (UniqueName: \"kubernetes.io/projected/bbbadd11-9ab2-4424-a66a-82f3d29423f9-kube-api-access-b9kqh\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:13:30.490990 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:30.490910    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmh6k"
Oct 30 21:13:30.490990 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:30.490952    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmh6k" event={"ID":"bbbadd11-9ab2-4424-a66a-82f3d29423f9","Type":"ContainerDied","Data":"42c0f7196fa45bff577380132b8ed51834f70db21f534f5e7b4fd962508ab9e5"}
Oct 30 21:13:30.491176 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:30.490998    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42c0f7196fa45bff577380132b8ed51834f70db21f534f5e7b4fd962508ab9e5"
Oct 30 21:13:34.265683 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:34.265650    5071 scope.go:122] "RemoveContainer" containerID="50e27ad7160c3ca4b4e990f2e2c7b2d34342c048ac53c4699df9a6a6af019e63"
Oct 30 21:13:34.274657 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:34.274629    5071 scope.go:122] "RemoveContainer" containerID="7abf74cb2b45af4d848c83165909547d4b5e9583158bd032c623461ab034b4a9"
Oct 30 21:13:34.283102 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:34.283082    5071 scope.go:122] "RemoveContainer" containerID="17ed1decfff6778053504b688c64dcaef2987e143baa52417821d3b91c3fb4cb"
Oct 30 21:13:39.213481 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:39.213383    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:13:39.566138 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:39.566102    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:13:39.838189 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:39.838107    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"]
Oct 30 21:13:39.838561 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:39.838548    5071 cpu_manager.go:414] "containerMap: removing container" podUID="bbbadd11-9ab2-4424-a66a-82f3d29423f9" containerName="revoke-keys"
Oct 30 21:13:39.838609 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:39.838562    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbbadd11-9ab2-4424-a66a-82f3d29423f9" containerName="revoke-keys"
Oct 30 21:13:39.838664 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:39.838624    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbbadd11-9ab2-4424-a66a-82f3d29423f9" containerName="revoke-keys"
Oct 30 21:13:39.842997 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:39.842976    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:13:39.869176 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:39.869140    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f15f7937-8890-478b-90ba-111e9b6d4a7d-istiod-ca-cert\") pod \"e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n\" (UID: \"f15f7937-8890-478b-90ba-111e9b6d4a7d\") " pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:13:39.869340 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:39.869186    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f15f7937-8890-478b-90ba-111e9b6d4a7d-workload-socket\") pod \"e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n\" (UID: \"f15f7937-8890-478b-90ba-111e9b6d4a7d\") " pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:13:39.869340 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:39.869206    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f15f7937-8890-478b-90ba-111e9b6d4a7d-istio-data\") pod \"e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n\" (UID: \"f15f7937-8890-478b-90ba-111e9b6d4a7d\") " pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:13:39.869340 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:39.869318    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f15f7937-8890-478b-90ba-111e9b6d4a7d-credential-socket\") pod \"e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n\" (UID: \"f15f7937-8890-478b-90ba-111e9b6d4a7d\") " pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:13:39.869523 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:39.869346    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f15f7937-8890-478b-90ba-111e9b6d4a7d-istio-envoy\") pod \"e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n\" (UID: \"f15f7937-8890-478b-90ba-111e9b6d4a7d\") " pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:13:39.869523 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:39.869368    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f15f7937-8890-478b-90ba-111e9b6d4a7d-istio-podinfo\") pod \"e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n\" (UID: \"f15f7937-8890-478b-90ba-111e9b6d4a7d\") " pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:13:39.869523 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:39.869462    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f15f7937-8890-478b-90ba-111e9b6d4a7d-istio-token\") pod \"e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n\" (UID: \"f15f7937-8890-478b-90ba-111e9b6d4a7d\") " pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:13:39.869523 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:39.869484    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f15f7937-8890-478b-90ba-111e9b6d4a7d-workload-certs\") pod \"e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n\" (UID: \"f15f7937-8890-478b-90ba-111e9b6d4a7d\") " pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:13:39.869523 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:39.869517    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq9v2\" (UniqueName: \"kubernetes.io/projected/f15f7937-8890-478b-90ba-111e9b6d4a7d-kube-api-access-xq9v2\") pod \"e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n\" (UID: \"f15f7937-8890-478b-90ba-111e9b6d4a7d\") " pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:13:39.973843 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:39.966191    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-dba0a6-openshift-default-dockercfg-n9g82\""
Oct 30 21:13:39.973843 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:39.970736    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"]
Oct 30 21:13:39.981844 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:39.978343    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f15f7937-8890-478b-90ba-111e9b6d4a7d-credential-socket\") pod \"e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n\" (UID: \"f15f7937-8890-478b-90ba-111e9b6d4a7d\") " pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:13:39.981844 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:39.978406    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f15f7937-8890-478b-90ba-111e9b6d4a7d-istio-envoy\") pod \"e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n\" (UID: \"f15f7937-8890-478b-90ba-111e9b6d4a7d\") " pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:13:39.981844 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:39.978438    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f15f7937-8890-478b-90ba-111e9b6d4a7d-istio-podinfo\") pod \"e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n\" (UID: \"f15f7937-8890-478b-90ba-111e9b6d4a7d\") " pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:13:39.981844 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:39.978496    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f15f7937-8890-478b-90ba-111e9b6d4a7d-istio-token\") pod \"e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n\" (UID: \"f15f7937-8890-478b-90ba-111e9b6d4a7d\") " pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:13:39.981844 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:39.978528    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f15f7937-8890-478b-90ba-111e9b6d4a7d-workload-certs\") pod \"e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n\" (UID: \"f15f7937-8890-478b-90ba-111e9b6d4a7d\") " pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:13:39.981844 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:39.978582    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq9v2\" (UniqueName: \"kubernetes.io/projected/f15f7937-8890-478b-90ba-111e9b6d4a7d-kube-api-access-xq9v2\") pod \"e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n\" (UID: \"f15f7937-8890-478b-90ba-111e9b6d4a7d\") " pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:13:39.981844 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:39.978632    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f15f7937-8890-478b-90ba-111e9b6d4a7d-istiod-ca-cert\") pod \"e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n\" (UID: \"f15f7937-8890-478b-90ba-111e9b6d4a7d\") " pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:13:39.981844 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:39.978684    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f15f7937-8890-478b-90ba-111e9b6d4a7d-workload-socket\") pod \"e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n\" (UID: \"f15f7937-8890-478b-90ba-111e9b6d4a7d\") " pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:13:39.981844 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:39.978714    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f15f7937-8890-478b-90ba-111e9b6d4a7d-istio-data\") pod \"e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n\" (UID: \"f15f7937-8890-478b-90ba-111e9b6d4a7d\") " pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:13:39.981844 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:39.979146    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f15f7937-8890-478b-90ba-111e9b6d4a7d-istio-data\") pod \"e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n\" (UID: \"f15f7937-8890-478b-90ba-111e9b6d4a7d\") " pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:13:39.981844 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:39.979884    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f15f7937-8890-478b-90ba-111e9b6d4a7d-credential-socket\") pod \"e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n\" (UID: \"f15f7937-8890-478b-90ba-111e9b6d4a7d\") " pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:13:40.020816 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:40.020592    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f15f7937-8890-478b-90ba-111e9b6d4a7d-workload-certs\") pod \"e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n\" (UID: \"f15f7937-8890-478b-90ba-111e9b6d4a7d\") " pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:13:40.021752 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:40.021705    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f15f7937-8890-478b-90ba-111e9b6d4a7d-istiod-ca-cert\") pod \"e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n\" (UID: \"f15f7937-8890-478b-90ba-111e9b6d4a7d\") " pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:13:40.021989 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:40.021957    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f15f7937-8890-478b-90ba-111e9b6d4a7d-workload-socket\") pod \"e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n\" (UID: \"f15f7937-8890-478b-90ba-111e9b6d4a7d\") " pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:13:40.027115 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:40.027083    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:13:40.028195 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:40.028155    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f15f7937-8890-478b-90ba-111e9b6d4a7d-istio-envoy\") pod \"e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n\" (UID: \"f15f7937-8890-478b-90ba-111e9b6d4a7d\") " pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:13:40.052430 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:40.052370    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f15f7937-8890-478b-90ba-111e9b6d4a7d-istio-podinfo\") pod \"e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n\" (UID: \"f15f7937-8890-478b-90ba-111e9b6d4a7d\") " pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:13:40.086169 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:40.086137    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f15f7937-8890-478b-90ba-111e9b6d4a7d-istio-token\") pod \"e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n\" (UID: \"f15f7937-8890-478b-90ba-111e9b6d4a7d\") " pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:13:40.088962 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:40.088902    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq9v2\" (UniqueName: \"kubernetes.io/projected/f15f7937-8890-478b-90ba-111e9b6d4a7d-kube-api-access-xq9v2\") pod \"e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n\" (UID: \"f15f7937-8890-478b-90ba-111e9b6d4a7d\") " pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:13:40.156113 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:40.156081    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:13:40.335641 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:40.335603    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:13:40.592047 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:40.592016    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"]
Oct 30 21:13:40.594147 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:13:40.594116    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf15f7937_8890_478b_90ba_111e9b6d4a7d.slice/crio-d03ed26c11bea8943fc3cd86f6165d0cfbb74b33a9dc23fbb0af5b59e2c27f3e WatchSource:0}: Error finding container d03ed26c11bea8943fc3cd86f6165d0cfbb74b33a9dc23fbb0af5b59e2c27f3e: Status 404 returned error can't find the container with id d03ed26c11bea8943fc3cd86f6165d0cfbb74b33a9dc23fbb0af5b59e2c27f3e
Oct 30 21:13:40.596327 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:40.596296    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:13:40.596401 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:40.596357    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:13:40.596401 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:40.596389    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:13:41.552286 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:41.552248    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n" event={"ID":"f15f7937-8890-478b-90ba-111e9b6d4a7d","Type":"ContainerStarted","Data":"1276d477511469cf40cd152e42ee8753b91fea724a7dbe5c5d0c4b5593238da9"}
Oct 30 21:13:41.552704 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:41.552305    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n" event={"ID":"f15f7937-8890-478b-90ba-111e9b6d4a7d","Type":"ContainerStarted","Data":"d03ed26c11bea8943fc3cd86f6165d0cfbb74b33a9dc23fbb0af5b59e2c27f3e"}
Oct 30 21:13:41.725350 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:41.725301    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n" podStartSLOduration=2.725286005 podStartE2EDuration="2.725286005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:41.722996516 +0000 UTC m=+2833.640018053" watchObservedRunningTime="2026-10-30 21:13:41.725286005 +0000 UTC m=+2833.642307543"
Oct 30 21:13:41.846243 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:41.846150    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:13:42.156441 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:42.156360    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:13:42.161530 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:42.161504    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:13:42.557579 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:42.557542    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:13:42.558667 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:42.558647    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:13:44.505100 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:44.505058    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:13:44.676039 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:44.676003    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:13:44.870139 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:44.870099    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:13:46.727500 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:13:46.727466    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:13:47.899275 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:47.899232    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:13:48.080200 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:48.080161    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:13:49.769729 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:49.769664    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:13:51.235082 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:51.235045    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:13:51.392269 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:51.392238    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:13:51.547680 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:51.547633    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:13:56.355979 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:56.355941    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:13:59.270005 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:13:59.269967    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:14:01.405240 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:01.405197    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:14:06.384812 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:06.384762    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:14:08.708285 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:08.708253    5071 generic.go:356] "Generic (PLEG): container finished" podID="f15f7937-8890-478b-90ba-111e9b6d4a7d" containerID="1276d477511469cf40cd152e42ee8753b91fea724a7dbe5c5d0c4b5593238da9" exitCode=0
Oct 30 21:14:08.708794 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:08.708441    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n" event={"ID":"f15f7937-8890-478b-90ba-111e9b6d4a7d","Type":"ContainerDied","Data":"1276d477511469cf40cd152e42ee8753b91fea724a7dbe5c5d0c4b5593238da9"}
Oct 30 21:14:08.708947 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:08.708914    5071 scope.go:122] "RemoveContainer" containerID="1276d477511469cf40cd152e42ee8753b91fea724a7dbe5c5d0c4b5593238da9"
Oct 30 21:14:08.710504 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:08.710429    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:14:08.710614 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:08.710587    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:14:08.710676 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:08.710661    5071 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:14:09.454603 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:09.454557    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:14:09.716849 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:09.716726    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n" event={"ID":"f15f7937-8890-478b-90ba-111e9b6d4a7d","Type":"ContainerStarted","Data":"98fbb27298731b86132b0a12fea0b305d9faab2cc79e3ab6d7fe59decce9a680"}
Oct 30 21:14:10.053886 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:10.053852    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:14:10.156326 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:10.156290    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:14:10.157780 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:10.157743    5071 patch_prober.go:29] interesting pod/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Oct 30 21:14:10.157944 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:10.157802    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n" podUID="f15f7937-8890-478b-90ba-111e9b6d4a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Oct 30 21:14:10.216767 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:10.216727    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:14:11.156431 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:11.156395    5071 patch_prober.go:29] interesting pod/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Oct 30 21:14:11.156860 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:11.156463    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n" podUID="f15f7937-8890-478b-90ba-111e9b6d4a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Oct 30 21:14:11.694482 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:11.694439    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:14:12.157360 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:12.157329    5071 patch_prober.go:29] interesting pod/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Oct 30 21:14:12.157909 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:12.157878    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n" podUID="f15f7937-8890-478b-90ba-111e9b6d4a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Oct 30 21:14:12.558159 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:12.558132    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:14:13.157388 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:13.157348    5071 patch_prober.go:29] interesting pod/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Oct 30 21:14:13.157803 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:13.157419    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n" podUID="f15f7937-8890-478b-90ba-111e9b6d4a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Oct 30 21:14:13.282083 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:13.282041    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:14:14.157299 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:14.157269    5071 patch_prober.go:29] interesting pod/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Oct 30 21:14:14.157548 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:14.157524    5071 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n" podUID="f15f7937-8890-478b-90ba-111e9b6d4a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Oct 30 21:14:15.161318 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:15.161288    5071 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:14:15.162337 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:15.162314    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:14:16.667632 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:16.667600    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:14:16.735728 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:14:16.735690    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:14:17.854925 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:17.854878    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:14:21.702412 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:21.702377    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:14:26.582188 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:26.582129    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:14:31.481620 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:31.481583    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:14:34.312349 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:34.312322    5071 scope.go:122] "RemoveContainer" containerID="5cc87a33fac2c3460604d86b01ba09221937e9c9df20bf5ea191bca0c6a68866"
Oct 30 21:14:34.321738 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:34.321717    5071 scope.go:122] "RemoveContainer" containerID="f50ece802a2a2397731254ce77a2072b104d40be7ab81b51915b49ecf4b8f3f0"
Oct 30 21:14:36.375259 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:36.375223    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:14:41.181038 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:41.180985    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:14:41.330125 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:41.330082    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:14:41.485470 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:41.485381    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:14:42.169646 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:42.169608    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:14:43.173933 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:43.173897    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:14:44.866749 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:44.866708    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:14:46.281563 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:46.281527    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:14:46.727514 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:14:46.727477    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:14:50.112653 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:50.112611    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:14:51.265514 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:51.265476    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:14:56.173143 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:56.173101    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:14:58.201897 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:58.201864    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a873e6d-jwgvb"]
Oct 30 21:14:58.205310 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:58.205292    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a873e6d-jwgvb"
Oct 30 21:14:58.244609 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:58.244563    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa49bbb7-63da-418b-9643-f4869e67393b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a873e6d-jwgvb\" (UID: \"aa49bbb7-63da-418b-9643-f4869e67393b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a873e6d-jwgvb"
Oct 30 21:14:58.244767 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:58.244686    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa49bbb7-63da-418b-9643-f4869e67393b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a873e6d-jwgvb\" (UID: \"aa49bbb7-63da-418b-9643-f4869e67393b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a873e6d-jwgvb"
Oct 30 21:14:58.244859 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:58.244786    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf9zf\" (UniqueName: \"kubernetes.io/projected/aa49bbb7-63da-418b-9643-f4869e67393b-kube-api-access-kf9zf\") pod \"maas-api-revoke-sub-6a873e6d-jwgvb\" (UID: \"aa49bbb7-63da-418b-9643-f4869e67393b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a873e6d-jwgvb"
Oct 30 21:14:58.258752 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:58.258725    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a873e6d-jwgvb"]
Oct 30 21:14:58.305086 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:58.305050    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:14:58.305310 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:58.305101    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:14:58.305750 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:58.305715    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:14:58.307800 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:58.307774    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 21:14:58.345548 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:58.345518    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf9zf\" (UniqueName: \"kubernetes.io/projected/aa49bbb7-63da-418b-9643-f4869e67393b-kube-api-access-kf9zf\") pod \"maas-api-revoke-sub-6a873e6d-jwgvb\" (UID: \"aa49bbb7-63da-418b-9643-f4869e67393b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a873e6d-jwgvb"
Oct 30 21:14:58.345686 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:58.345562    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa49bbb7-63da-418b-9643-f4869e67393b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a873e6d-jwgvb\" (UID: \"aa49bbb7-63da-418b-9643-f4869e67393b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a873e6d-jwgvb"
Oct 30 21:14:58.345686 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:58.345609    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa49bbb7-63da-418b-9643-f4869e67393b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a873e6d-jwgvb\" (UID: \"aa49bbb7-63da-418b-9643-f4869e67393b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a873e6d-jwgvb"
Oct 30 21:14:58.346264 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:58.346243    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa49bbb7-63da-418b-9643-f4869e67393b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a873e6d-jwgvb\" (UID: \"aa49bbb7-63da-418b-9643-f4869e67393b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a873e6d-jwgvb"
Oct 30 21:14:58.348089 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:58.348070    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa49bbb7-63da-418b-9643-f4869e67393b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a873e6d-jwgvb\" (UID: \"aa49bbb7-63da-418b-9643-f4869e67393b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a873e6d-jwgvb"
Oct 30 21:14:58.404324 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:58.404290    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf9zf\" (UniqueName: \"kubernetes.io/projected/aa49bbb7-63da-418b-9643-f4869e67393b-kube-api-access-kf9zf\") pod \"maas-api-revoke-sub-6a873e6d-jwgvb\" (UID: \"aa49bbb7-63da-418b-9643-f4869e67393b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a873e6d-jwgvb"
Oct 30 21:14:58.517749 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:58.517664    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a873e6d-jwgvb"
Oct 30 21:14:58.888747 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:58.888719    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a873e6d-jwgvb"]
Oct 30 21:14:58.890737 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:14:58.890708    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa49bbb7_63da_418b_9643_f4869e67393b.slice/crio-107368f1336312069334165a0d4003a07be8b006f481e35a1a294d3ac5e2c414 WatchSource:0}: Error finding container 107368f1336312069334165a0d4003a07be8b006f481e35a1a294d3ac5e2c414: Status 404 returned error can't find the container with id 107368f1336312069334165a0d4003a07be8b006f481e35a1a294d3ac5e2c414
Oct 30 21:14:59.022484 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:14:59.022452    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a873e6d-jwgvb" event={"ID":"aa49bbb7-63da-418b-9643-f4869e67393b","Type":"ContainerStarted","Data":"107368f1336312069334165a0d4003a07be8b006f481e35a1a294d3ac5e2c414"}
Oct 30 21:15:00.029670 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:00.029580    5071 generic.go:356] "Generic (PLEG): container finished" podID="aa49bbb7-63da-418b-9643-f4869e67393b" containerID="b9eee5b284d7a258af53f603261914fa93cd9b9f8a1789d54fdbb2fe0e6f14e9" exitCode=0
Oct 30 21:15:00.030099 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:00.029667    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a873e6d-jwgvb" event={"ID":"aa49bbb7-63da-418b-9643-f4869e67393b","Type":"ContainerDied","Data":"b9eee5b284d7a258af53f603261914fa93cd9b9f8a1789d54fdbb2fe0e6f14e9"}
Oct 30 21:15:00.201124 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:00.201088    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-dba0a6-2988991bhwlh"]
Oct 30 21:15:00.204782 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:00.204763    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-dba0a6-2988991bhwlh"
Oct 30 21:15:00.260708 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:00.260674    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-dba0a6-2988991bhwlh"]
Oct 30 21:15:00.263365 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:00.263335    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s779\" (UniqueName: \"kubernetes.io/projected/696860c9-ebf1-4151-886c-00f8b4ab05e6-kube-api-access-6s779\") pod \"maas-api-key-cleanup-e2e-worker-single-main-dba0a6-2988991bhwlh\" (UID: \"696860c9-ebf1-4151-886c-00f8b4ab05e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-dba0a6-2988991bhwlh"
Oct 30 21:15:00.263522 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:00.263408    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/696860c9-ebf1-4151-886c-00f8b4ab05e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-dba0a6-2988991bhwlh\" (UID: \"696860c9-ebf1-4151-886c-00f8b4ab05e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-dba0a6-2988991bhwlh"
Oct 30 21:15:00.364220 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:00.364174    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s779\" (UniqueName: \"kubernetes.io/projected/696860c9-ebf1-4151-886c-00f8b4ab05e6-kube-api-access-6s779\") pod \"maas-api-key-cleanup-e2e-worker-single-main-dba0a6-2988991bhwlh\" (UID: \"696860c9-ebf1-4151-886c-00f8b4ab05e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-dba0a6-2988991bhwlh"
Oct 30 21:15:00.364392 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:00.364229    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/696860c9-ebf1-4151-886c-00f8b4ab05e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-dba0a6-2988991bhwlh\" (UID: \"696860c9-ebf1-4151-886c-00f8b4ab05e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-dba0a6-2988991bhwlh"
Oct 30 21:15:00.364859 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:00.364837    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/696860c9-ebf1-4151-886c-00f8b4ab05e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-dba0a6-2988991bhwlh\" (UID: \"696860c9-ebf1-4151-886c-00f8b4ab05e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-dba0a6-2988991bhwlh"
Oct 30 21:15:00.422634 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:00.422604    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s779\" (UniqueName: \"kubernetes.io/projected/696860c9-ebf1-4151-886c-00f8b4ab05e6-kube-api-access-6s779\") pod \"maas-api-key-cleanup-e2e-worker-single-main-dba0a6-2988991bhwlh\" (UID: \"696860c9-ebf1-4151-886c-00f8b4ab05e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-dba0a6-2988991bhwlh"
Oct 30 21:15:00.517406 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:00.517374    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-dba0a6-2988991bhwlh"
Oct 30 21:15:00.891263 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:00.891241    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-dba0a6-2988991bhwlh"]
Oct 30 21:15:00.893335 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:15:00.893305    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod696860c9_ebf1_4151_886c_00f8b4ab05e6.slice/crio-ea535a04dac87de9faf9d0ac7fc05e0a242ee52e75dc8e97c0937e6d65f838bf WatchSource:0}: Error finding container ea535a04dac87de9faf9d0ac7fc05e0a242ee52e75dc8e97c0937e6d65f838bf: Status 404 returned error can't find the container with id ea535a04dac87de9faf9d0ac7fc05e0a242ee52e75dc8e97c0937e6d65f838bf
Oct 30 21:15:01.036702 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:01.036669    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-dba0a6-2988991bhwlh" event={"ID":"696860c9-ebf1-4151-886c-00f8b4ab05e6","Type":"ContainerStarted","Data":"ea535a04dac87de9faf9d0ac7fc05e0a242ee52e75dc8e97c0937e6d65f838bf"}
Oct 30 21:15:01.098422 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:01.098388    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:15:01.183510 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:01.183489    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a873e6d-jwgvb"
Oct 30 21:15:01.272586 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:01.272555    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa49bbb7-63da-418b-9643-f4869e67393b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa49bbb7-63da-418b-9643-f4869e67393b-maas-api-openshift-ca\") pod \"aa49bbb7-63da-418b-9643-f4869e67393b\" (UID: \"aa49bbb7-63da-418b-9643-f4869e67393b\") "
Oct 30 21:15:01.272753 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:01.272597    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa49bbb7-63da-418b-9643-f4869e67393b-kube-api-access-kf9zf\" (UniqueName: \"kubernetes.io/projected/aa49bbb7-63da-418b-9643-f4869e67393b-kube-api-access-kf9zf\") pod \"aa49bbb7-63da-418b-9643-f4869e67393b\" (UID: \"aa49bbb7-63da-418b-9643-f4869e67393b\") "
Oct 30 21:15:01.272753 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:01.272699    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa49bbb7-63da-418b-9643-f4869e67393b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa49bbb7-63da-418b-9643-f4869e67393b-maas-api-xks-ca\") pod \"aa49bbb7-63da-418b-9643-f4869e67393b\" (UID: \"aa49bbb7-63da-418b-9643-f4869e67393b\") "
Oct 30 21:15:01.272986 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:01.272961    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa49bbb7-63da-418b-9643-f4869e67393b-maas-api-openshift-ca" pod "aa49bbb7-63da-418b-9643-f4869e67393b" (UID: "aa49bbb7-63da-418b-9643-f4869e67393b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:01.273148 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:01.273124    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa49bbb7-63da-418b-9643-f4869e67393b-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:01.275664 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:01.275642    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa49bbb7-63da-418b-9643-f4869e67393b-maas-api-xks-ca" pod "aa49bbb7-63da-418b-9643-f4869e67393b" (UID: "aa49bbb7-63da-418b-9643-f4869e67393b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:15:01.275760 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:01.275713    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa49bbb7-63da-418b-9643-f4869e67393b-kube-api-access-kf9zf" pod "aa49bbb7-63da-418b-9643-f4869e67393b" (UID: "aa49bbb7-63da-418b-9643-f4869e67393b"). InnerVolumeSpecName "kube-api-access-kf9zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:01.373720 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:01.373685    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf9zf\" (UniqueName: \"kubernetes.io/projected/aa49bbb7-63da-418b-9643-f4869e67393b-kube-api-access-kf9zf\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:01.373720 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:01.373714    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa49bbb7-63da-418b-9643-f4869e67393b-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:02.043707 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:02.043669    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a873e6d-jwgvb" event={"ID":"aa49bbb7-63da-418b-9643-f4869e67393b","Type":"ContainerDied","Data":"107368f1336312069334165a0d4003a07be8b006f481e35a1a294d3ac5e2c414"}
Oct 30 21:15:02.044258 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:02.043720    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="107368f1336312069334165a0d4003a07be8b006f481e35a1a294d3ac5e2c414"
Oct 30 21:15:02.044258 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:02.043677    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a873e6d-jwgvb"
Oct 30 21:15:02.045412 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:02.045384    5071 generic.go:356] "Generic (PLEG): container finished" podID="696860c9-ebf1-4151-886c-00f8b4ab05e6" containerID="253f70450587ecdb2e6aab5f0f3ad5ebd895d215ca602ff350a8805a1df73509" exitCode=60
Oct 30 21:15:02.045546 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:02.045471    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-dba0a6-2988991bhwlh" event={"ID":"696860c9-ebf1-4151-886c-00f8b4ab05e6","Type":"ContainerDied","Data":"253f70450587ecdb2e6aab5f0f3ad5ebd895d215ca602ff350a8805a1df73509"}
Oct 30 21:15:02.045680 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:02.045663    5071 scope.go:122] "RemoveContainer" containerID="253f70450587ecdb2e6aab5f0f3ad5ebd895d215ca602ff350a8805a1df73509"
Oct 30 21:15:02.292021 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:02.291977    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:15:02.515305 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:02.515271    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:15:03.052906 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:03.052871    5071 generic.go:356] "Generic (PLEG): container finished" podID="696860c9-ebf1-4151-886c-00f8b4ab05e6" containerID="7ab0e327e9365da90f596af7f9363629d6dcacb0e8f47b07127349653e0e17f0" exitCode=60
Oct 30 21:15:03.053338 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:03.052949    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-dba0a6-2988991bhwlh" event={"ID":"696860c9-ebf1-4151-886c-00f8b4ab05e6","Type":"ContainerDied","Data":"7ab0e327e9365da90f596af7f9363629d6dcacb0e8f47b07127349653e0e17f0"}
Oct 30 21:15:03.053338 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:03.053011    5071 scope.go:122] "RemoveContainer" containerID="253f70450587ecdb2e6aab5f0f3ad5ebd895d215ca602ff350a8805a1df73509"
Oct 30 21:15:03.053338 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:03.053255    5071 scope.go:122] "RemoveContainer" containerID="7ab0e327e9365da90f596af7f9363629d6dcacb0e8f47b07127349653e0e17f0"
Oct 30 21:15:03.053543 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:15:03.053519    5071 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-dba0a6-2988991bhwlh_odh-ai-gateway-infra(696860c9-ebf1-4151-886c-00f8b4ab05e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-dba0a6-2988991bhwlh" podUID="696860c9-ebf1-4151-886c-00f8b4ab05e6"
Oct 30 21:15:04.061361 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:04.061331    5071 scope.go:122] "RemoveContainer" containerID="7ab0e327e9365da90f596af7f9363629d6dcacb0e8f47b07127349653e0e17f0"
Oct 30 21:15:04.061900 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:15:04.061592    5071 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-dba0a6-2988991bhwlh_odh-ai-gateway-infra(696860c9-ebf1-4151-886c-00f8b4ab05e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-dba0a6-2988991bhwlh" podUID="696860c9-ebf1-4151-886c-00f8b4ab05e6"
Oct 30 21:15:05.966946 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:05.966911    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:15:09.333683 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:09.333587    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:15:09.863235 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:09.863201    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dba0a6-c78hw"]
Oct 30 21:15:09.863676 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:09.863662    5071 cpu_manager.go:414] "containerMap: removing container" podUID="aa49bbb7-63da-418b-9643-f4869e67393b" containerName="revoke-keys"
Oct 30 21:15:09.863723 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:09.863678    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa49bbb7-63da-418b-9643-f4869e67393b" containerName="revoke-keys"
Oct 30 21:15:09.863780 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:09.863772    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa49bbb7-63da-418b-9643-f4869e67393b" containerName="revoke-keys"
Oct 30 21:15:09.868516 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:09.868497    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dba0a6-c78hw"
Oct 30 21:15:09.925095 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:09.925057    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dba0a6-c78hw"]
Oct 30 21:15:09.962153 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:09.962121    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51886550-9d42-41ea-a9e1-43bd91097092-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dba0a6-c78hw\" (UID: \"51886550-9d42-41ea-a9e1-43bd91097092\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dba0a6-c78hw"
Oct 30 21:15:09.962322 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:09.962187    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51886550-9d42-41ea-a9e1-43bd91097092-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dba0a6-c78hw\" (UID: \"51886550-9d42-41ea-a9e1-43bd91097092\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dba0a6-c78hw"
Oct 30 21:15:09.962322 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:09.962301    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nwg9\" (UniqueName: \"kubernetes.io/projected/51886550-9d42-41ea-a9e1-43bd91097092-kube-api-access-5nwg9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dba0a6-c78hw\" (UID: \"51886550-9d42-41ea-a9e1-43bd91097092\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dba0a6-c78hw"
Oct 30 21:15:09.971378 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:09.971355    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:15:10.063972 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:10.063917    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nwg9\" (UniqueName: \"kubernetes.io/projected/51886550-9d42-41ea-a9e1-43bd91097092-kube-api-access-5nwg9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dba0a6-c78hw\" (UID: \"51886550-9d42-41ea-a9e1-43bd91097092\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dba0a6-c78hw"
Oct 30 21:15:10.064167 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:10.064030    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51886550-9d42-41ea-a9e1-43bd91097092-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dba0a6-c78hw\" (UID: \"51886550-9d42-41ea-a9e1-43bd91097092\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dba0a6-c78hw"
Oct 30 21:15:10.064167 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:10.064089    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51886550-9d42-41ea-a9e1-43bd91097092-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dba0a6-c78hw\" (UID: \"51886550-9d42-41ea-a9e1-43bd91097092\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dba0a6-c78hw"
Oct 30 21:15:10.064733 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:10.064715    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51886550-9d42-41ea-a9e1-43bd91097092-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dba0a6-c78hw\" (UID: \"51886550-9d42-41ea-a9e1-43bd91097092\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dba0a6-c78hw"
Oct 30 21:15:10.066599 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:10.066574    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51886550-9d42-41ea-a9e1-43bd91097092-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dba0a6-c78hw\" (UID: \"51886550-9d42-41ea-a9e1-43bd91097092\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dba0a6-c78hw"
Oct 30 21:15:10.122169 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:10.122101    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nwg9\" (UniqueName: \"kubernetes.io/projected/51886550-9d42-41ea-a9e1-43bd91097092-kube-api-access-5nwg9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-dba0a6-c78hw\" (UID: \"51886550-9d42-41ea-a9e1-43bd91097092\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dba0a6-c78hw"
Oct 30 21:15:10.183779 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:10.183747    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dba0a6-c78hw"
Oct 30 21:15:10.556660 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:10.556625    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dba0a6-c78hw"]
Oct 30 21:15:10.558508 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:15:10.558475    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51886550_9d42_41ea_a9e1_43bd91097092.slice/crio-4fad9f23be6d5e9e22c1478f9e32ffade76138f0e6bc2412237d87fc8c9e8a56 WatchSource:0}: Error finding container 4fad9f23be6d5e9e22c1478f9e32ffade76138f0e6bc2412237d87fc8c9e8a56: Status 404 returned error can't find the container with id 4fad9f23be6d5e9e22c1478f9e32ffade76138f0e6bc2412237d87fc8c9e8a56
Oct 30 21:15:11.103203 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:11.103118    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dba0a6-c78hw" event={"ID":"51886550-9d42-41ea-a9e1-43bd91097092","Type":"ContainerStarted","Data":"984a03a50c6f359fc256a9abd9e464b81381961d75398b4f7791fbcdfcbfa725"}
Oct 30 21:15:11.103203 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:11.103173    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dba0a6-c78hw" event={"ID":"51886550-9d42-41ea-a9e1-43bd91097092","Type":"ContainerStarted","Data":"4fad9f23be6d5e9e22c1478f9e32ffade76138f0e6bc2412237d87fc8c9e8a56"}
Oct 30 21:15:11.111520 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:11.111484    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:15:11.223235 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:11.223188    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dba0a6-c78hw" podStartSLOduration=2.223173088 podStartE2EDuration="2.223173088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:11.220912814 +0000 UTC m=+2923.137934346" watchObservedRunningTime="2026-10-30 21:15:11.223173088 +0000 UTC m=+2923.140194623"
Oct 30 21:15:11.240364 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:11.240329    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:15:12.110863 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:12.110805    5071 generic.go:356] "Generic (PLEG): container finished" podID="51886550-9d42-41ea-a9e1-43bd91097092" containerID="984a03a50c6f359fc256a9abd9e464b81381961d75398b4f7791fbcdfcbfa725" exitCode=0
Oct 30 21:15:12.111345 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:12.110873    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dba0a6-c78hw" event={"ID":"51886550-9d42-41ea-a9e1-43bd91097092","Type":"ContainerDied","Data":"984a03a50c6f359fc256a9abd9e464b81381961d75398b4f7791fbcdfcbfa725"}
Oct 30 21:15:13.266255 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:13.266224    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dba0a6-c78hw"
Oct 30 21:15:13.394797 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:13.394707    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51886550-9d42-41ea-a9e1-43bd91097092-kube-api-access-5nwg9\" (UniqueName: \"kubernetes.io/projected/51886550-9d42-41ea-a9e1-43bd91097092-kube-api-access-5nwg9\") pod \"51886550-9d42-41ea-a9e1-43bd91097092\" (UID: \"51886550-9d42-41ea-a9e1-43bd91097092\") "
Oct 30 21:15:13.394797 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:13.394776    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51886550-9d42-41ea-a9e1-43bd91097092-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51886550-9d42-41ea-a9e1-43bd91097092-maas-api-xks-ca\") pod \"51886550-9d42-41ea-a9e1-43bd91097092\" (UID: \"51886550-9d42-41ea-a9e1-43bd91097092\") "
Oct 30 21:15:13.395081 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:13.394890    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51886550-9d42-41ea-a9e1-43bd91097092-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51886550-9d42-41ea-a9e1-43bd91097092-maas-api-openshift-ca\") pod \"51886550-9d42-41ea-a9e1-43bd91097092\" (UID: \"51886550-9d42-41ea-a9e1-43bd91097092\") "
Oct 30 21:15:13.395276 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:13.395245    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51886550-9d42-41ea-a9e1-43bd91097092-maas-api-openshift-ca" pod "51886550-9d42-41ea-a9e1-43bd91097092" (UID: "51886550-9d42-41ea-a9e1-43bd91097092"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:13.397033 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:13.397000    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51886550-9d42-41ea-a9e1-43bd91097092-kube-api-access-5nwg9" pod "51886550-9d42-41ea-a9e1-43bd91097092" (UID: "51886550-9d42-41ea-a9e1-43bd91097092"). InnerVolumeSpecName "kube-api-access-5nwg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:13.397176 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:13.397155    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51886550-9d42-41ea-a9e1-43bd91097092-maas-api-xks-ca" pod "51886550-9d42-41ea-a9e1-43bd91097092" (UID: "51886550-9d42-41ea-a9e1-43bd91097092"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:15:13.496143 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:13.496092    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51886550-9d42-41ea-a9e1-43bd91097092-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:13.496143 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:13.496139    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nwg9\" (UniqueName: \"kubernetes.io/projected/51886550-9d42-41ea-a9e1-43bd91097092-kube-api-access-5nwg9\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:13.496143 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:13.496153    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51886550-9d42-41ea-a9e1-43bd91097092-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:14.123954 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:14.123923    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dba0a6-c78hw"
Oct 30 21:15:14.124127 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:14.123959    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dba0a6-c78hw" event={"ID":"51886550-9d42-41ea-a9e1-43bd91097092","Type":"ContainerDied","Data":"4fad9f23be6d5e9e22c1478f9e32ffade76138f0e6bc2412237d87fc8c9e8a56"}
Oct 30 21:15:14.124127 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:14.124038    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fad9f23be6d5e9e22c1478f9e32ffade76138f0e6bc2412237d87fc8c9e8a56"
Oct 30 21:15:14.478557 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:14.478468    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:15:16.186288 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:16.186248    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:15:16.726758 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:15:16.726722    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:15:17.194934 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:17.194904    5071 scope.go:122] "RemoveContainer" containerID="7ab0e327e9365da90f596af7f9363629d6dcacb0e8f47b07127349653e0e17f0"
Oct 30 21:15:18.147097 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:18.147064    5071 generic.go:356] "Generic (PLEG): container finished" podID="696860c9-ebf1-4151-886c-00f8b4ab05e6" containerID="3e73ab119f655ea9fce0bdc73cb6ffc15e3eef7dc190fb75568ce9066ca500ef" exitCode=60
Oct 30 21:15:18.147247 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:18.147136    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-dba0a6-2988991bhwlh" event={"ID":"696860c9-ebf1-4151-886c-00f8b4ab05e6","Type":"ContainerDied","Data":"3e73ab119f655ea9fce0bdc73cb6ffc15e3eef7dc190fb75568ce9066ca500ef"}
Oct 30 21:15:18.147247 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:18.147197    5071 scope.go:122] "RemoveContainer" containerID="7ab0e327e9365da90f596af7f9363629d6dcacb0e8f47b07127349653e0e17f0"
Oct 30 21:15:18.147538 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:18.147521    5071 scope.go:122] "RemoveContainer" containerID="3e73ab119f655ea9fce0bdc73cb6ffc15e3eef7dc190fb75568ce9066ca500ef"
Oct 30 21:15:18.147795 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:15:18.147773    5071 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-dba0a6-2988991bhwlh_odh-ai-gateway-infra(696860c9-ebf1-4151-886c-00f8b4ab05e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-dba0a6-2988991bhwlh" podUID="696860c9-ebf1-4151-886c-00f8b4ab05e6"
Oct 30 21:15:19.273815 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:19.273780    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-dba0a6-2988991bhwlh"]
Oct 30 21:15:19.417922 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:19.417887    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-dba0a6-2988991bhwlh"
Oct 30 21:15:19.557360 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:19.557329    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/696860c9-ebf1-4151-886c-00f8b4ab05e6-kube-api-access-6s779\" (UniqueName: \"kubernetes.io/projected/696860c9-ebf1-4151-886c-00f8b4ab05e6-kube-api-access-6s779\") pod \"696860c9-ebf1-4151-886c-00f8b4ab05e6\" (UID: \"696860c9-ebf1-4151-886c-00f8b4ab05e6\") "
Oct 30 21:15:19.557544 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:19.557449    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/696860c9-ebf1-4151-886c-00f8b4ab05e6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/696860c9-ebf1-4151-886c-00f8b4ab05e6-maas-api-service-ca\") pod \"696860c9-ebf1-4151-886c-00f8b4ab05e6\" (UID: \"696860c9-ebf1-4151-886c-00f8b4ab05e6\") "
Oct 30 21:15:19.557884 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:19.557847    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/696860c9-ebf1-4151-886c-00f8b4ab05e6-maas-api-service-ca" pod "696860c9-ebf1-4151-886c-00f8b4ab05e6" (UID: "696860c9-ebf1-4151-886c-00f8b4ab05e6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:19.559685 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:19.559660    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/696860c9-ebf1-4151-886c-00f8b4ab05e6-kube-api-access-6s779" pod "696860c9-ebf1-4151-886c-00f8b4ab05e6" (UID: "696860c9-ebf1-4151-886c-00f8b4ab05e6"). InnerVolumeSpecName "kube-api-access-6s779". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:19.659220 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:19.659183    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s779\" (UniqueName: \"kubernetes.io/projected/696860c9-ebf1-4151-886c-00f8b4ab05e6-kube-api-access-6s779\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:19.659220 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:19.659214    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/696860c9-ebf1-4151-886c-00f8b4ab05e6-maas-api-service-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:20.161918 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:20.161890    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-dba0a6-2988991bhwlh"
Oct 30 21:15:20.161918 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:20.161902    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-dba0a6-2988991bhwlh" event={"ID":"696860c9-ebf1-4151-886c-00f8b4ab05e6","Type":"ContainerDied","Data":"ea535a04dac87de9faf9d0ac7fc05e0a242ee52e75dc8e97c0937e6d65f838bf"}
Oct 30 21:15:20.162152 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:20.161983    5071 scope.go:122] "RemoveContainer" containerID="3e73ab119f655ea9fce0bdc73cb6ffc15e3eef7dc190fb75568ce9066ca500ef"
Oct 30 21:15:20.330875 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:20.330839    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-dba0a6-2988991bhwlh"]
Oct 30 21:15:20.338246 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:20.338212    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-dba0a6-2988991bhwlh"]
Oct 30 21:15:20.515960 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:20.515862    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:15:21.045632 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:21.045536    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:15:21.251434 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:21.251369    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:15:22.200311 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:22.200277    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="696860c9-ebf1-4151-886c-00f8b4ab05e6" path="/var/lib/kubelet/pods/696860c9-ebf1-4151-886c-00f8b4ab05e6/volumes"
Oct 30 21:15:25.177185 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:25.177144    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:15:25.556377 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:25.556349    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:15:25.829891 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:25.829780    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:15:26.188935 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:26.188854    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dba0a6-c78hw"]
Oct 30 21:15:26.238333 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:26.238303    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-dba0a6-c78hw"]
Oct 30 21:15:26.597014 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:26.596979    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:15:28.201784 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:28.201747    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51886550-9d42-41ea-a9e1-43bd91097092" path="/var/lib/kubelet/pods/51886550-9d42-41ea-a9e1-43bd91097092/volumes"
Oct 30 21:15:30.050131 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:30.050095    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f192083-pdf82"]
Oct 30 21:15:30.099307 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:30.099274    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f192083-pdf82"]
Oct 30 21:15:30.201963 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:30.201919    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="974db133-15d8-49a3-acaa-23ec86daf784" path="/var/lib/kubelet/pods/974db133-15d8-49a3-acaa-23ec86daf784/volumes"
Oct 30 21:15:33.295199 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:33.295160    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:15:33.502203 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:33.502159    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:15:33.616243 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:33.616168    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"]
Oct 30 21:15:33.616439 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:33.616418    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n" podUID="f15f7937-8890-478b-90ba-111e9b6d4a7d" containerName="istio-proxy" containerID="cri-o://98fbb27298731b86132b0a12fea0b305d9faab2cc79e3ab6d7fe59decce9a680" gracePeriod=30
Oct 30 21:15:33.655790 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:33.655754    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:15:34.346006 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:34.345976    5071 scope.go:122] "RemoveContainer" containerID="9df0582b16cf767056936d304feaad03013e70abd13d0d859543f0efc505bd1e"
Oct 30 21:15:38.881392 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:38.881339    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:15:38.965647 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:38.965619    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f15f7937-8890-478b-90ba-111e9b6d4a7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f15f7937-8890-478b-90ba-111e9b6d4a7d-istio-envoy\") pod \"f15f7937-8890-478b-90ba-111e9b6d4a7d\" (UID: \"f15f7937-8890-478b-90ba-111e9b6d4a7d\") "
Oct 30 21:15:38.965802 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:38.965666    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f15f7937-8890-478b-90ba-111e9b6d4a7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f15f7937-8890-478b-90ba-111e9b6d4a7d-istio-data\") pod \"f15f7937-8890-478b-90ba-111e9b6d4a7d\" (UID: \"f15f7937-8890-478b-90ba-111e9b6d4a7d\") "
Oct 30 21:15:38.965802 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:38.965699    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f15f7937-8890-478b-90ba-111e9b6d4a7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f15f7937-8890-478b-90ba-111e9b6d4a7d-istiod-ca-cert\") pod \"f15f7937-8890-478b-90ba-111e9b6d4a7d\" (UID: \"f15f7937-8890-478b-90ba-111e9b6d4a7d\") "
Oct 30 21:15:38.965802 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:38.965726    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f15f7937-8890-478b-90ba-111e9b6d4a7d-istio-token\" (UniqueName: \"kubernetes.io/projected/f15f7937-8890-478b-90ba-111e9b6d4a7d-istio-token\") pod \"f15f7937-8890-478b-90ba-111e9b6d4a7d\" (UID: \"f15f7937-8890-478b-90ba-111e9b6d4a7d\") "
Oct 30 21:15:38.965802 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:38.965787    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f15f7937-8890-478b-90ba-111e9b6d4a7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f15f7937-8890-478b-90ba-111e9b6d4a7d-istio-podinfo\") pod \"f15f7937-8890-478b-90ba-111e9b6d4a7d\" (UID: \"f15f7937-8890-478b-90ba-111e9b6d4a7d\") "
Oct 30 21:15:38.966047 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:38.965885    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f15f7937-8890-478b-90ba-111e9b6d4a7d-kube-api-access-xq9v2\" (UniqueName: \"kubernetes.io/projected/f15f7937-8890-478b-90ba-111e9b6d4a7d-kube-api-access-xq9v2\") pod \"f15f7937-8890-478b-90ba-111e9b6d4a7d\" (UID: \"f15f7937-8890-478b-90ba-111e9b6d4a7d\") "
Oct 30 21:15:38.966047 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:38.965919    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f15f7937-8890-478b-90ba-111e9b6d4a7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f15f7937-8890-478b-90ba-111e9b6d4a7d-workload-certs\") pod \"f15f7937-8890-478b-90ba-111e9b6d4a7d\" (UID: \"f15f7937-8890-478b-90ba-111e9b6d4a7d\") "
Oct 30 21:15:38.966047 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:38.965964    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f15f7937-8890-478b-90ba-111e9b6d4a7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f15f7937-8890-478b-90ba-111e9b6d4a7d-credential-socket\") pod \"f15f7937-8890-478b-90ba-111e9b6d4a7d\" (UID: \"f15f7937-8890-478b-90ba-111e9b6d4a7d\") "
Oct 30 21:15:38.966047 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:38.966030    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f15f7937-8890-478b-90ba-111e9b6d4a7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f15f7937-8890-478b-90ba-111e9b6d4a7d-workload-socket\") pod \"f15f7937-8890-478b-90ba-111e9b6d4a7d\" (UID: \"f15f7937-8890-478b-90ba-111e9b6d4a7d\") "
Oct 30 21:15:38.966281 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:38.966037    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f15f7937-8890-478b-90ba-111e9b6d4a7d-istio-data" pod "f15f7937-8890-478b-90ba-111e9b6d4a7d" (UID: "f15f7937-8890-478b-90ba-111e9b6d4a7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:38.966337 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:38.966298    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f15f7937-8890-478b-90ba-111e9b6d4a7d-credential-socket" pod "f15f7937-8890-478b-90ba-111e9b6d4a7d" (UID: "f15f7937-8890-478b-90ba-111e9b6d4a7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:38.966337 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:38.966207    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f15f7937-8890-478b-90ba-111e9b6d4a7d-istiod-ca-cert" pod "f15f7937-8890-478b-90ba-111e9b6d4a7d" (UID: "f15f7937-8890-478b-90ba-111e9b6d4a7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:38.966449 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:38.966415    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f15f7937-8890-478b-90ba-111e9b6d4a7d-workload-certs" pod "f15f7937-8890-478b-90ba-111e9b6d4a7d" (UID: "f15f7937-8890-478b-90ba-111e9b6d4a7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:38.966564 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:38.966507    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f15f7937-8890-478b-90ba-111e9b6d4a7d-workload-socket" pod "f15f7937-8890-478b-90ba-111e9b6d4a7d" (UID: "f15f7937-8890-478b-90ba-111e9b6d4a7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:38.966564 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:38.966576    5071 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f15f7937-8890-478b-90ba-111e9b6d4a7d-credential-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:38.966854 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:38.966597    5071 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f15f7937-8890-478b-90ba-111e9b6d4a7d-istio-data\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:38.966854 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:38.966613    5071 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f15f7937-8890-478b-90ba-111e9b6d4a7d-istiod-ca-cert\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:38.968053 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:38.968024    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f15f7937-8890-478b-90ba-111e9b6d4a7d-istio-envoy" pod "f15f7937-8890-478b-90ba-111e9b6d4a7d" (UID: "f15f7937-8890-478b-90ba-111e9b6d4a7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:38.968253 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:38.968228    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f15f7937-8890-478b-90ba-111e9b6d4a7d-istio-podinfo" pod "f15f7937-8890-478b-90ba-111e9b6d4a7d" (UID: "f15f7937-8890-478b-90ba-111e9b6d4a7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:15:38.968325 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:38.968272    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f15f7937-8890-478b-90ba-111e9b6d4a7d-istio-token" pod "f15f7937-8890-478b-90ba-111e9b6d4a7d" (UID: "f15f7937-8890-478b-90ba-111e9b6d4a7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:38.968365 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:38.968334    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f15f7937-8890-478b-90ba-111e9b6d4a7d-kube-api-access-xq9v2" pod "f15f7937-8890-478b-90ba-111e9b6d4a7d" (UID: "f15f7937-8890-478b-90ba-111e9b6d4a7d"). InnerVolumeSpecName "kube-api-access-xq9v2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:39.067458 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:39.067425    5071 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f15f7937-8890-478b-90ba-111e9b6d4a7d-workload-socket\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:39.067458 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:39.067454    5071 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f15f7937-8890-478b-90ba-111e9b6d4a7d-istio-envoy\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:39.067717 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:39.067469    5071 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f15f7937-8890-478b-90ba-111e9b6d4a7d-istio-token\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:39.067717 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:39.067477    5071 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f15f7937-8890-478b-90ba-111e9b6d4a7d-istio-podinfo\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:39.067717 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:39.067486    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq9v2\" (UniqueName: \"kubernetes.io/projected/f15f7937-8890-478b-90ba-111e9b6d4a7d-kube-api-access-xq9v2\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:39.067717 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:39.067495    5071 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f15f7937-8890-478b-90ba-111e9b6d4a7d-workload-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:39.272843 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:39.272741    5071 generic.go:356] "Generic (PLEG): container finished" podID="f15f7937-8890-478b-90ba-111e9b6d4a7d" containerID="98fbb27298731b86132b0a12fea0b305d9faab2cc79e3ab6d7fe59decce9a680" exitCode=0
Oct 30 21:15:39.272843 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:39.272794    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"
Oct 30 21:15:39.272843 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:39.272811    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n" event={"ID":"f15f7937-8890-478b-90ba-111e9b6d4a7d","Type":"ContainerDied","Data":"98fbb27298731b86132b0a12fea0b305d9faab2cc79e3ab6d7fe59decce9a680"}
Oct 30 21:15:39.273096 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:39.272881    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n" event={"ID":"f15f7937-8890-478b-90ba-111e9b6d4a7d","Type":"ContainerDied","Data":"d03ed26c11bea8943fc3cd86f6165d0cfbb74b33a9dc23fbb0af5b59e2c27f3e"}
Oct 30 21:15:39.273096 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:39.272903    5071 scope.go:122] "RemoveContainer" containerID="98fbb27298731b86132b0a12fea0b305d9faab2cc79e3ab6d7fe59decce9a680"
Oct 30 21:15:39.282975 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:39.282953    5071 scope.go:122] "RemoveContainer" containerID="1276d477511469cf40cd152e42ee8753b91fea724a7dbe5c5d0c4b5593238da9"
Oct 30 21:15:39.291205 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:39.291190    5071 scope.go:122] "RemoveContainer" containerID="98fbb27298731b86132b0a12fea0b305d9faab2cc79e3ab6d7fe59decce9a680"
Oct 30 21:15:39.291458 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:15:39.291442    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98fbb27298731b86132b0a12fea0b305d9faab2cc79e3ab6d7fe59decce9a680\": container with ID starting with 98fbb27298731b86132b0a12fea0b305d9faab2cc79e3ab6d7fe59decce9a680 not found: ID does not exist" containerID="98fbb27298731b86132b0a12fea0b305d9faab2cc79e3ab6d7fe59decce9a680"
Oct 30 21:15:39.291498 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:39.291467    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98fbb27298731b86132b0a12fea0b305d9faab2cc79e3ab6d7fe59decce9a680"} err="failed to get container status \"98fbb27298731b86132b0a12fea0b305d9faab2cc79e3ab6d7fe59decce9a680\": rpc error: code = NotFound desc = could not find container \"98fbb27298731b86132b0a12fea0b305d9faab2cc79e3ab6d7fe59decce9a680\": container with ID starting with 98fbb27298731b86132b0a12fea0b305d9faab2cc79e3ab6d7fe59decce9a680 not found: ID does not exist"
Oct 30 21:15:39.291498 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:39.291484    5071 scope.go:122] "RemoveContainer" containerID="1276d477511469cf40cd152e42ee8753b91fea724a7dbe5c5d0c4b5593238da9"
Oct 30 21:15:39.291708 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:15:39.291695    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1276d477511469cf40cd152e42ee8753b91fea724a7dbe5c5d0c4b5593238da9\": container with ID starting with 1276d477511469cf40cd152e42ee8753b91fea724a7dbe5c5d0c4b5593238da9 not found: ID does not exist" containerID="1276d477511469cf40cd152e42ee8753b91fea724a7dbe5c5d0c4b5593238da9"
Oct 30 21:15:39.291749 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:39.291713    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1276d477511469cf40cd152e42ee8753b91fea724a7dbe5c5d0c4b5593238da9"} err="failed to get container status \"1276d477511469cf40cd152e42ee8753b91fea724a7dbe5c5d0c4b5593238da9\": rpc error: code = NotFound desc = could not find container \"1276d477511469cf40cd152e42ee8753b91fea724a7dbe5c5d0c4b5593238da9\": container with ID starting with 1276d477511469cf40cd152e42ee8753b91fea724a7dbe5c5d0c4b5593238da9 not found: ID does not exist"
Oct 30 21:15:39.440753 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:39.440719    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"]
Oct 30 21:15:39.447639 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:39.447613    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-dba0a6-openshift-default-7c555dc9b9c569n"]
Oct 30 21:15:40.200584 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:40.200552    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f15f7937-8890-478b-90ba-111e9b6d4a7d" path="/var/lib/kubelet/pods/f15f7937-8890-478b-90ba-111e9b6d4a7d/volumes"
Oct 30 21:15:41.070334 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:41.070298    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:15:42.517349 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:42.517301    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:15:45.161457 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:45.161413    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:15:45.555989 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:45.555954    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:15:46.745525 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:15:46.745479    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:15:50.447124 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:50.447084    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:15:51.100089 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:51.100050    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:15:52.477070 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:15:52.477023    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:16:04.217135 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:04.217096    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:16:06.306746 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:06.306711    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:16:07.224527 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:07.224482    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:16:08.331520 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:08.331491    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:16:11.053644 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:11.053606    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q9fbn"]
Oct 30 21:16:11.102916 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:11.102884    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q9fbn"]
Oct 30 21:16:12.201512 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:12.201475    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="777b0a45-e119-42c9-a6ee-8fb4ed9446c9" path="/var/lib/kubelet/pods/777b0a45-e119-42c9-a6ee-8fb4ed9446c9/volumes"
Oct 30 21:16:13.047833 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:13.047797    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:16:16.727615 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:16:16.727577    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:16:25.453612 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:25.453581    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:16:28.878633 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:28.878601    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8qvb_91fe9e06-0c30-4000-bddb-2ba89cd048cc/console-operator/2.log"
Oct 30 21:16:28.886592 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:28.886568    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q4f4h_80cae79e-b60f-4efe-bb58-e77753a457e5/ovn-acl-logging/0.log"
Oct 30 21:16:28.895233 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:28.895211    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8qvb_91fe9e06-0c30-4000-bddb-2ba89cd048cc/console-operator/2.log"
Oct 30 21:16:28.902568 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:28.902548    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q4f4h_80cae79e-b60f-4efe-bb58-e77753a457e5/ovn-acl-logging/0.log"
Oct 30 21:16:30.076294 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:30.076262    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:16:34.373256 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:34.373224    5071 scope.go:122] "RemoveContainer" containerID="bdb787102e882bdf44f43fe82ced4509f7e27b27a2e8d1b93c8559d148e878d1"
Oct 30 21:16:42.877423 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:42.877331    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:16:45.472463 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:45.472425    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:16:46.727307 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:16:46.727265    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:16:51.829679 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:51.829636    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:16:56.284808 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:56.284771    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:16:56.456186 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:56.456144    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:16:57.190058 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:57.190028    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z7hvt"]
Oct 30 21:16:57.190546 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:57.190527    5071 cpu_manager.go:414] "containerMap: removing container" podUID="696860c9-ebf1-4151-886c-00f8b4ab05e6" containerName="cleanup"
Oct 30 21:16:57.190675 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:57.190548    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="696860c9-ebf1-4151-886c-00f8b4ab05e6" containerName="cleanup"
Oct 30 21:16:57.190675 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:57.190578    5071 cpu_manager.go:414] "containerMap: removing container" podUID="51886550-9d42-41ea-a9e1-43bd91097092" containerName="revoke-keys"
Oct 30 21:16:57.190675 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:57.190587    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51886550-9d42-41ea-a9e1-43bd91097092" containerName="revoke-keys"
Oct 30 21:16:57.190675 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:57.190605    5071 cpu_manager.go:414] "containerMap: removing container" podUID="f15f7937-8890-478b-90ba-111e9b6d4a7d" containerName="istio-proxy"
Oct 30 21:16:57.190675 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:57.190615    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f15f7937-8890-478b-90ba-111e9b6d4a7d" containerName="istio-proxy"
Oct 30 21:16:57.190675 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:57.190635    5071 cpu_manager.go:414] "containerMap: removing container" podUID="696860c9-ebf1-4151-886c-00f8b4ab05e6" containerName="cleanup"
Oct 30 21:16:57.190675 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:57.190649    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="696860c9-ebf1-4151-886c-00f8b4ab05e6" containerName="cleanup"
Oct 30 21:16:57.191069 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:57.190681    5071 cpu_manager.go:414] "containerMap: removing container" podUID="f15f7937-8890-478b-90ba-111e9b6d4a7d" containerName="istio-proxy"
Oct 30 21:16:57.191069 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:57.190689    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f15f7937-8890-478b-90ba-111e9b6d4a7d" containerName="istio-proxy"
Oct 30 21:16:57.191069 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:57.190803    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="f15f7937-8890-478b-90ba-111e9b6d4a7d" containerName="istio-proxy"
Oct 30 21:16:57.191069 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:57.190816    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="696860c9-ebf1-4151-886c-00f8b4ab05e6" containerName="cleanup"
Oct 30 21:16:57.191069 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:57.190847    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="51886550-9d42-41ea-a9e1-43bd91097092" containerName="revoke-keys"
Oct 30 21:16:57.191069 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:57.190858    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="696860c9-ebf1-4151-886c-00f8b4ab05e6" containerName="cleanup"
Oct 30 21:16:57.191069 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:57.190869    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="696860c9-ebf1-4151-886c-00f8b4ab05e6" containerName="cleanup"
Oct 30 21:16:57.194103 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:57.194082    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z7hvt"
Oct 30 21:16:57.252555 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:57.252521    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z7hvt"]
Oct 30 21:16:57.298478 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:57.298442    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:16:57.298891 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:57.298480    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:16:57.299412 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:57.299392    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:16:57.299489 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:57.299465    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 21:16:57.342560 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:57.342526    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbvs8\" (UniqueName: \"kubernetes.io/projected/05076e73-051d-479a-990e-93bf067ede16-kube-api-access-jbvs8\") pod \"maas-api-revoke-sub-452b6d7b-z7hvt\" (UID: \"05076e73-051d-479a-990e-93bf067ede16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z7hvt"
Oct 30 21:16:57.342736 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:57.342689    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05076e73-051d-479a-990e-93bf067ede16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-z7hvt\" (UID: \"05076e73-051d-479a-990e-93bf067ede16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z7hvt"
Oct 30 21:16:57.342736 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:57.342722    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05076e73-051d-479a-990e-93bf067ede16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-z7hvt\" (UID: \"05076e73-051d-479a-990e-93bf067ede16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z7hvt"
Oct 30 21:16:57.444331 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:57.444252    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05076e73-051d-479a-990e-93bf067ede16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-z7hvt\" (UID: \"05076e73-051d-479a-990e-93bf067ede16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z7hvt"
Oct 30 21:16:57.444331 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:57.444287    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05076e73-051d-479a-990e-93bf067ede16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-z7hvt\" (UID: \"05076e73-051d-479a-990e-93bf067ede16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z7hvt"
Oct 30 21:16:57.444578 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:57.444343    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbvs8\" (UniqueName: \"kubernetes.io/projected/05076e73-051d-479a-990e-93bf067ede16-kube-api-access-jbvs8\") pod \"maas-api-revoke-sub-452b6d7b-z7hvt\" (UID: \"05076e73-051d-479a-990e-93bf067ede16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z7hvt"
Oct 30 21:16:57.445071 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:57.445043    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05076e73-051d-479a-990e-93bf067ede16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-z7hvt\" (UID: \"05076e73-051d-479a-990e-93bf067ede16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z7hvt"
Oct 30 21:16:57.446716 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:57.446696    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05076e73-051d-479a-990e-93bf067ede16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-z7hvt\" (UID: \"05076e73-051d-479a-990e-93bf067ede16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z7hvt"
Oct 30 21:16:57.502499 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:57.502467    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbvs8\" (UniqueName: \"kubernetes.io/projected/05076e73-051d-479a-990e-93bf067ede16-kube-api-access-jbvs8\") pod \"maas-api-revoke-sub-452b6d7b-z7hvt\" (UID: \"05076e73-051d-479a-990e-93bf067ede16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z7hvt"
Oct 30 21:16:57.506843 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:57.506794    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z7hvt"
Oct 30 21:16:57.879068 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:57.879039    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z7hvt"]
Oct 30 21:16:57.880539 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:16:57.880511    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05076e73_051d_479a_990e_93bf067ede16.slice/crio-204aa75cc5305bc6b41927a892cee5c6c715eb712427743e29d306bbc3de9287 WatchSource:0}: Error finding container 204aa75cc5305bc6b41927a892cee5c6c715eb712427743e29d306bbc3de9287: Status 404 returned error can't find the container with id 204aa75cc5305bc6b41927a892cee5c6c715eb712427743e29d306bbc3de9287
Oct 30 21:16:58.644062 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:58.644029    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:16:58.771050 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:58.771014    5071 generic.go:356] "Generic (PLEG): container finished" podID="05076e73-051d-479a-990e-93bf067ede16" containerID="725fe8f0e08b86f2694083d9030500c890394088e8169a2f8b3b18a6033ed6a9" exitCode=0
Oct 30 21:16:58.771215 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:58.771086    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z7hvt" event={"ID":"05076e73-051d-479a-990e-93bf067ede16","Type":"ContainerDied","Data":"725fe8f0e08b86f2694083d9030500c890394088e8169a2f8b3b18a6033ed6a9"}
Oct 30 21:16:58.771215 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:16:58.771136    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z7hvt" event={"ID":"05076e73-051d-479a-990e-93bf067ede16","Type":"ContainerStarted","Data":"204aa75cc5305bc6b41927a892cee5c6c715eb712427743e29d306bbc3de9287"}
Oct 30 21:17:00.028153 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:00.028121    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z7hvt"
Oct 30 21:17:00.192494 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:00.192399    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05076e73-051d-479a-990e-93bf067ede16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05076e73-051d-479a-990e-93bf067ede16-maas-api-xks-ca\") pod \"05076e73-051d-479a-990e-93bf067ede16\" (UID: \"05076e73-051d-479a-990e-93bf067ede16\") "
Oct 30 21:17:00.192792 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:00.192767    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05076e73-051d-479a-990e-93bf067ede16-kube-api-access-jbvs8\" (UniqueName: \"kubernetes.io/projected/05076e73-051d-479a-990e-93bf067ede16-kube-api-access-jbvs8\") pod \"05076e73-051d-479a-990e-93bf067ede16\" (UID: \"05076e73-051d-479a-990e-93bf067ede16\") "
Oct 30 21:17:00.192999 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:00.192980    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05076e73-051d-479a-990e-93bf067ede16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05076e73-051d-479a-990e-93bf067ede16-maas-api-openshift-ca\") pod \"05076e73-051d-479a-990e-93bf067ede16\" (UID: \"05076e73-051d-479a-990e-93bf067ede16\") "
Oct 30 21:17:00.196184 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:00.196160    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05076e73-051d-479a-990e-93bf067ede16-maas-api-openshift-ca" pod "05076e73-051d-479a-990e-93bf067ede16" (UID: "05076e73-051d-479a-990e-93bf067ede16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:17:00.211185 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:00.211138    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05076e73-051d-479a-990e-93bf067ede16-maas-api-xks-ca" pod "05076e73-051d-479a-990e-93bf067ede16" (UID: "05076e73-051d-479a-990e-93bf067ede16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:17:00.213370 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:00.213334    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05076e73-051d-479a-990e-93bf067ede16-kube-api-access-jbvs8" pod "05076e73-051d-479a-990e-93bf067ede16" (UID: "05076e73-051d-479a-990e-93bf067ede16"). InnerVolumeSpecName "kube-api-access-jbvs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:17:00.297539 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:00.297475    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05076e73-051d-479a-990e-93bf067ede16-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:00.297804 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:00.297783    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbvs8\" (UniqueName: \"kubernetes.io/projected/05076e73-051d-479a-990e-93bf067ede16-kube-api-access-jbvs8\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:00.297957 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:00.297940    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05076e73-051d-479a-990e-93bf067ede16-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:00.821854 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:00.818145    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z7hvt" event={"ID":"05076e73-051d-479a-990e-93bf067ede16","Type":"ContainerDied","Data":"204aa75cc5305bc6b41927a892cee5c6c715eb712427743e29d306bbc3de9287"}
Oct 30 21:17:00.821854 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:00.818210    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="204aa75cc5305bc6b41927a892cee5c6c715eb712427743e29d306bbc3de9287"
Oct 30 21:17:00.821854 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:00.818276    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z7hvt"
Oct 30 21:17:01.975994 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:01.975953    5071 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cv4kg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:17:01.976709 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:01.976656    5071 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cv4kg" podUID="9be8fbd1-6cb8-4718-8cfe-a45894b5d33e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:10.104452 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:10.104392    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:17:14.565698 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:14.565648    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:17:16.868847 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:17:16.868073    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:17:28.867423 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:28.867384    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:17:32.986895 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:32.986854    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t7zm7"]
Oct 30 21:17:32.987337 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:32.987322    5071 cpu_manager.go:414] "containerMap: removing container" podUID="05076e73-051d-479a-990e-93bf067ede16" containerName="revoke-keys"
Oct 30 21:17:32.987394 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:32.987339    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05076e73-051d-479a-990e-93bf067ede16" containerName="revoke-keys"
Oct 30 21:17:32.987394 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:32.987362    5071 cpu_manager.go:414] "containerMap: removing container" podUID="696860c9-ebf1-4151-886c-00f8b4ab05e6" containerName="cleanup"
Oct 30 21:17:32.987394 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:32.987367    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="696860c9-ebf1-4151-886c-00f8b4ab05e6" containerName="cleanup"
Oct 30 21:17:32.987490 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:32.987444    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="05076e73-051d-479a-990e-93bf067ede16" containerName="revoke-keys"
Oct 30 21:17:32.987490 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:32.987456    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="f15f7937-8890-478b-90ba-111e9b6d4a7d" containerName="istio-proxy"
Oct 30 21:17:32.990815 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:32.990796    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t7zm7"
Oct 30 21:17:33.046968 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:33.046941    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:17:33.095782 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:33.095752    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:17:33.095782 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:33.095780    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:17:33.096029 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:33.095804    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 21:17:33.098173 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:33.098156    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t7zm7"]
Oct 30 21:17:33.128804 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:33.128774    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j94r\" (UniqueName: \"kubernetes.io/projected/7687aa1f-92bb-4fca-8a33-515fa3a2241f-kube-api-access-7j94r\") pod \"maas-api-revoke-sub-3e31a506-t7zm7\" (UID: \"7687aa1f-92bb-4fca-8a33-515fa3a2241f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t7zm7"
Oct 30 21:17:33.128980 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:33.128811    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7687aa1f-92bb-4fca-8a33-515fa3a2241f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-t7zm7\" (UID: \"7687aa1f-92bb-4fca-8a33-515fa3a2241f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t7zm7"
Oct 30 21:17:33.128980 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:33.128953    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7687aa1f-92bb-4fca-8a33-515fa3a2241f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-t7zm7\" (UID: \"7687aa1f-92bb-4fca-8a33-515fa3a2241f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t7zm7"
Oct 30 21:17:33.230451 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:33.230414    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j94r\" (UniqueName: \"kubernetes.io/projected/7687aa1f-92bb-4fca-8a33-515fa3a2241f-kube-api-access-7j94r\") pod \"maas-api-revoke-sub-3e31a506-t7zm7\" (UID: \"7687aa1f-92bb-4fca-8a33-515fa3a2241f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t7zm7"
Oct 30 21:17:33.230657 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:33.230462    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7687aa1f-92bb-4fca-8a33-515fa3a2241f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-t7zm7\" (UID: \"7687aa1f-92bb-4fca-8a33-515fa3a2241f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t7zm7"
Oct 30 21:17:33.230657 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:33.230521    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7687aa1f-92bb-4fca-8a33-515fa3a2241f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-t7zm7\" (UID: \"7687aa1f-92bb-4fca-8a33-515fa3a2241f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t7zm7"
Oct 30 21:17:33.231308 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:33.231279    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7687aa1f-92bb-4fca-8a33-515fa3a2241f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-t7zm7\" (UID: \"7687aa1f-92bb-4fca-8a33-515fa3a2241f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t7zm7"
Oct 30 21:17:33.233144 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:33.233125    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7687aa1f-92bb-4fca-8a33-515fa3a2241f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-t7zm7\" (UID: \"7687aa1f-92bb-4fca-8a33-515fa3a2241f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t7zm7"
Oct 30 21:17:33.251623 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:33.251555    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:17:33.292299 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:33.292268    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j94r\" (UniqueName: \"kubernetes.io/projected/7687aa1f-92bb-4fca-8a33-515fa3a2241f-kube-api-access-7j94r\") pod \"maas-api-revoke-sub-3e31a506-t7zm7\" (UID: \"7687aa1f-92bb-4fca-8a33-515fa3a2241f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t7zm7"
Oct 30 21:17:33.303862 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:33.303807    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t7zm7"
Oct 30 21:17:33.689845 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:33.689770    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t7zm7"]
Oct 30 21:17:33.701404 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:17:33.701374    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7687aa1f_92bb_4fca_8a33_515fa3a2241f.slice/crio-c26e7cdfe2d179e29ae648032ab699a73d897767b4ebf165302e2f66d276c163 WatchSource:0}: Error finding container c26e7cdfe2d179e29ae648032ab699a73d897767b4ebf165302e2f66d276c163: Status 404 returned error can't find the container with id c26e7cdfe2d179e29ae648032ab699a73d897767b4ebf165302e2f66d276c163
Oct 30 21:17:34.032371 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:34.032284    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t7zm7" event={"ID":"7687aa1f-92bb-4fca-8a33-515fa3a2241f","Type":"ContainerStarted","Data":"c26e7cdfe2d179e29ae648032ab699a73d897767b4ebf165302e2f66d276c163"}
Oct 30 21:17:34.399072 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:34.399026    5071 scope.go:122] "RemoveContainer" containerID="a29eb97753954fc10d388f40c67e2ea2d52650aab9d32fa953651bda337ae27b"
Oct 30 21:17:34.412085 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:34.412038    5071 scope.go:122] "RemoveContainer" containerID="2f404813e989e4c141da77dd43bd2299c4a59c607ee5be9f2a8c98b41adc1c0c"
Oct 30 21:17:35.082341 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:35.082303    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t7zm7" event={"ID":"7687aa1f-92bb-4fca-8a33-515fa3a2241f","Type":"ContainerStarted","Data":"cb398cb9a6bd62ae58d22bae7ae02fbee58f57c8376397898d615ab9ffbade50"}
Oct 30 21:17:36.103849 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:36.099054    5071 generic.go:356] "Generic (PLEG): container finished" podID="7687aa1f-92bb-4fca-8a33-515fa3a2241f" containerID="cb398cb9a6bd62ae58d22bae7ae02fbee58f57c8376397898d615ab9ffbade50" exitCode=0
Oct 30 21:17:36.103849 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:36.099230    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t7zm7" event={"ID":"7687aa1f-92bb-4fca-8a33-515fa3a2241f","Type":"ContainerDied","Data":"cb398cb9a6bd62ae58d22bae7ae02fbee58f57c8376397898d615ab9ffbade50"}
Oct 30 21:17:37.254051 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:37.254027    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t7zm7"
Oct 30 21:17:37.325528 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:37.325495    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7687aa1f-92bb-4fca-8a33-515fa3a2241f-kube-api-access-7j94r\" (UniqueName: \"kubernetes.io/projected/7687aa1f-92bb-4fca-8a33-515fa3a2241f-kube-api-access-7j94r\") pod \"7687aa1f-92bb-4fca-8a33-515fa3a2241f\" (UID: \"7687aa1f-92bb-4fca-8a33-515fa3a2241f\") "
Oct 30 21:17:37.325695 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:37.325551    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7687aa1f-92bb-4fca-8a33-515fa3a2241f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7687aa1f-92bb-4fca-8a33-515fa3a2241f-maas-api-openshift-ca\") pod \"7687aa1f-92bb-4fca-8a33-515fa3a2241f\" (UID: \"7687aa1f-92bb-4fca-8a33-515fa3a2241f\") "
Oct 30 21:17:37.325695 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:37.325655    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7687aa1f-92bb-4fca-8a33-515fa3a2241f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7687aa1f-92bb-4fca-8a33-515fa3a2241f-maas-api-xks-ca\") pod \"7687aa1f-92bb-4fca-8a33-515fa3a2241f\" (UID: \"7687aa1f-92bb-4fca-8a33-515fa3a2241f\") "
Oct 30 21:17:37.325871 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:37.325843    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7687aa1f-92bb-4fca-8a33-515fa3a2241f-maas-api-openshift-ca" pod "7687aa1f-92bb-4fca-8a33-515fa3a2241f" (UID: "7687aa1f-92bb-4fca-8a33-515fa3a2241f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:17:37.326502 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:37.326463    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7687aa1f-92bb-4fca-8a33-515fa3a2241f-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:37.327904 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:37.327879    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7687aa1f-92bb-4fca-8a33-515fa3a2241f-kube-api-access-7j94r" pod "7687aa1f-92bb-4fca-8a33-515fa3a2241f" (UID: "7687aa1f-92bb-4fca-8a33-515fa3a2241f"). InnerVolumeSpecName "kube-api-access-7j94r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:17:37.327904 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:37.327877    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7687aa1f-92bb-4fca-8a33-515fa3a2241f-maas-api-xks-ca" pod "7687aa1f-92bb-4fca-8a33-515fa3a2241f" (UID: "7687aa1f-92bb-4fca-8a33-515fa3a2241f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:17:37.427276 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:37.427195    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7687aa1f-92bb-4fca-8a33-515fa3a2241f-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:37.427276 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:37.427226    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j94r\" (UniqueName: \"kubernetes.io/projected/7687aa1f-92bb-4fca-8a33-515fa3a2241f-kube-api-access-7j94r\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:37.652435 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:37.652390    5071 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-149.us-west-1.compute.internal" podUID="9d9f66ca5c88277c8c76e3090f63988d" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:38.110597 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:38.110558    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t7zm7" event={"ID":"7687aa1f-92bb-4fca-8a33-515fa3a2241f","Type":"ContainerDied","Data":"c26e7cdfe2d179e29ae648032ab699a73d897767b4ebf165302e2f66d276c163"}
Oct 30 21:17:38.110597 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:38.110588    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t7zm7"
Oct 30 21:17:38.110933 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:38.110617    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c26e7cdfe2d179e29ae648032ab699a73d897767b4ebf165302e2f66d276c163"
Oct 30 21:17:43.251703 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:43.251667    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:17:45.758478 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:45.758445    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:17:45.834576 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:45.834538    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:17:46.876096 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:17:46.876049    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:17:53.288851 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:53.288520    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:17:55.945141 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:17:55.945103    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:18:02.051637 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:02.051591    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cswg"]
Oct 30 21:18:02.102103 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:02.102066    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-6cswg"]
Oct 30 21:18:02.200095 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:02.200059    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="297b7ce4-811b-4092-bd6b-2fb22b9f7cea" path="/var/lib/kubelet/pods/297b7ce4-811b-4092-bd6b-2fb22b9f7cea/volumes"
Oct 30 21:18:05.771489 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:05.771456    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6bee289-2qbmm"]
Oct 30 21:18:05.771948 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:05.771933    5071 cpu_manager.go:414] "containerMap: removing container" podUID="7687aa1f-92bb-4fca-8a33-515fa3a2241f" containerName="revoke-keys"
Oct 30 21:18:05.772001 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:05.771949    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7687aa1f-92bb-4fca-8a33-515fa3a2241f" containerName="revoke-keys"
Oct 30 21:18:05.772058 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:05.772049    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="7687aa1f-92bb-4fca-8a33-515fa3a2241f" containerName="revoke-keys"
Oct 30 21:18:05.775459 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:05.775441    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6bee289-2qbmm"
Oct 30 21:18:05.829935 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:05.829907    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:18:05.846273 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:05.846244    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e673006-6fdf-4ac9-8186-afe96fe4f5df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6bee289-2qbmm\" (UID: \"7e673006-6fdf-4ac9-8186-afe96fe4f5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6bee289-2qbmm"
Oct 30 21:18:05.846430 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:05.846281    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e673006-6fdf-4ac9-8186-afe96fe4f5df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6bee289-2qbmm\" (UID: \"7e673006-6fdf-4ac9-8186-afe96fe4f5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6bee289-2qbmm"
Oct 30 21:18:05.846430 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:05.846395    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dngq\" (UniqueName: \"kubernetes.io/projected/7e673006-6fdf-4ac9-8186-afe96fe4f5df-kube-api-access-6dngq\") pod \"maas-api-revoke-sub-d6bee289-2qbmm\" (UID: \"7e673006-6fdf-4ac9-8186-afe96fe4f5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6bee289-2qbmm"
Oct 30 21:18:05.878718 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:05.878690    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:18:05.878718 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:05.878690    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:18:05.878919 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:05.878728    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-78djw\""
Oct 30 21:18:05.881290 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:05.881270    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6bee289-2qbmm"]
Oct 30 21:18:05.947894 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:05.947860    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e673006-6fdf-4ac9-8186-afe96fe4f5df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6bee289-2qbmm\" (UID: \"7e673006-6fdf-4ac9-8186-afe96fe4f5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6bee289-2qbmm"
Oct 30 21:18:05.948080 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:05.947906    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e673006-6fdf-4ac9-8186-afe96fe4f5df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6bee289-2qbmm\" (UID: \"7e673006-6fdf-4ac9-8186-afe96fe4f5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6bee289-2qbmm"
Oct 30 21:18:05.948080 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:05.947952    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dngq\" (UniqueName: \"kubernetes.io/projected/7e673006-6fdf-4ac9-8186-afe96fe4f5df-kube-api-access-6dngq\") pod \"maas-api-revoke-sub-d6bee289-2qbmm\" (UID: \"7e673006-6fdf-4ac9-8186-afe96fe4f5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6bee289-2qbmm"
Oct 30 21:18:05.948440 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:05.948419    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e673006-6fdf-4ac9-8186-afe96fe4f5df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6bee289-2qbmm\" (UID: \"7e673006-6fdf-4ac9-8186-afe96fe4f5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6bee289-2qbmm"
Oct 30 21:18:05.950449 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:05.950427    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e673006-6fdf-4ac9-8186-afe96fe4f5df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6bee289-2qbmm\" (UID: \"7e673006-6fdf-4ac9-8186-afe96fe4f5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6bee289-2qbmm"
Oct 30 21:18:06.005716 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:06.005693    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dngq\" (UniqueName: \"kubernetes.io/projected/7e673006-6fdf-4ac9-8186-afe96fe4f5df-kube-api-access-6dngq\") pod \"maas-api-revoke-sub-d6bee289-2qbmm\" (UID: \"7e673006-6fdf-4ac9-8186-afe96fe4f5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6bee289-2qbmm"
Oct 30 21:18:06.088658 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:06.088630    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6bee289-2qbmm"
Oct 30 21:18:06.461491 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:06.461468    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6bee289-2qbmm"]
Oct 30 21:18:06.463249 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:18:06.463221    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e673006_6fdf_4ac9_8186_afe96fe4f5df.slice/crio-22eee11f9eadbe08bc4535d71313944e2776f34e90f3ffec247413b7bec35216 WatchSource:0}: Error finding container 22eee11f9eadbe08bc4535d71313944e2776f34e90f3ffec247413b7bec35216: Status 404 returned error can't find the container with id 22eee11f9eadbe08bc4535d71313944e2776f34e90f3ffec247413b7bec35216
Oct 30 21:18:07.332092 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:07.332051    5071 generic.go:356] "Generic (PLEG): container finished" podID="7e673006-6fdf-4ac9-8186-afe96fe4f5df" containerID="fee0d9c3a15fa10aa2cf160e4ff97c2605e5a34af6155d0cc2765a29e8edfcd7" exitCode=0
Oct 30 21:18:07.332480 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:07.332111    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6bee289-2qbmm" event={"ID":"7e673006-6fdf-4ac9-8186-afe96fe4f5df","Type":"ContainerDied","Data":"fee0d9c3a15fa10aa2cf160e4ff97c2605e5a34af6155d0cc2765a29e8edfcd7"}
Oct 30 21:18:07.332480 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:07.332174    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6bee289-2qbmm" event={"ID":"7e673006-6fdf-4ac9-8186-afe96fe4f5df","Type":"ContainerStarted","Data":"22eee11f9eadbe08bc4535d71313944e2776f34e90f3ffec247413b7bec35216"}
Oct 30 21:18:08.481742 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:08.481719    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6bee289-2qbmm"
Oct 30 21:18:08.574234 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:08.574207    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e673006-6fdf-4ac9-8186-afe96fe4f5df-kube-api-access-6dngq\" (UniqueName: \"kubernetes.io/projected/7e673006-6fdf-4ac9-8186-afe96fe4f5df-kube-api-access-6dngq\") pod \"7e673006-6fdf-4ac9-8186-afe96fe4f5df\" (UID: \"7e673006-6fdf-4ac9-8186-afe96fe4f5df\") "
Oct 30 21:18:08.574395 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:08.574321    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e673006-6fdf-4ac9-8186-afe96fe4f5df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e673006-6fdf-4ac9-8186-afe96fe4f5df-maas-api-openshift-ca\") pod \"7e673006-6fdf-4ac9-8186-afe96fe4f5df\" (UID: \"7e673006-6fdf-4ac9-8186-afe96fe4f5df\") "
Oct 30 21:18:08.574395 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:08.574380    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e673006-6fdf-4ac9-8186-afe96fe4f5df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e673006-6fdf-4ac9-8186-afe96fe4f5df-maas-api-xks-ca\") pod \"7e673006-6fdf-4ac9-8186-afe96fe4f5df\" (UID: \"7e673006-6fdf-4ac9-8186-afe96fe4f5df\") "
Oct 30 21:18:08.574703 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:08.574674    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e673006-6fdf-4ac9-8186-afe96fe4f5df-maas-api-openshift-ca" pod "7e673006-6fdf-4ac9-8186-afe96fe4f5df" (UID: "7e673006-6fdf-4ac9-8186-afe96fe4f5df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:08.576581 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:08.576555    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e673006-6fdf-4ac9-8186-afe96fe4f5df-maas-api-xks-ca" pod "7e673006-6fdf-4ac9-8186-afe96fe4f5df" (UID: "7e673006-6fdf-4ac9-8186-afe96fe4f5df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:18:08.576581 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:08.576566    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e673006-6fdf-4ac9-8186-afe96fe4f5df-kube-api-access-6dngq" pod "7e673006-6fdf-4ac9-8186-afe96fe4f5df" (UID: "7e673006-6fdf-4ac9-8186-afe96fe4f5df"). InnerVolumeSpecName "kube-api-access-6dngq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:08.675897 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:08.675810    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e673006-6fdf-4ac9-8186-afe96fe4f5df-maas-api-xks-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:08.675897 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:08.675852    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dngq\" (UniqueName: \"kubernetes.io/projected/7e673006-6fdf-4ac9-8186-afe96fe4f5df-kube-api-access-6dngq\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:08.675897 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:08.675863    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e673006-6fdf-4ac9-8186-afe96fe4f5df-maas-api-openshift-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:09.345179 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:09.345148    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6bee289-2qbmm"
Oct 30 21:18:09.345359 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:09.345181    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6bee289-2qbmm" event={"ID":"7e673006-6fdf-4ac9-8186-afe96fe4f5df","Type":"ContainerDied","Data":"22eee11f9eadbe08bc4535d71313944e2776f34e90f3ffec247413b7bec35216"}
Oct 30 21:18:09.345359 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:09.345222    5071 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22eee11f9eadbe08bc4535d71313944e2776f34e90f3ffec247413b7bec35216"
Oct 30 21:18:16.729413 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:18:16.729374    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:18:18.588835 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:18.588783    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:18:19.119112 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:19.119078    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-84b5f84777-txqd7"]
Oct 30 21:18:19.119518 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:19.119505    5071 cpu_manager.go:414] "containerMap: removing container" podUID="7e673006-6fdf-4ac9-8186-afe96fe4f5df" containerName="revoke-keys"
Oct 30 21:18:19.119566 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:19.119519    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e673006-6fdf-4ac9-8186-afe96fe4f5df" containerName="revoke-keys"
Oct 30 21:18:19.119621 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:19.119611    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e673006-6fdf-4ac9-8186-afe96fe4f5df" containerName="revoke-keys"
Oct 30 21:18:19.122865 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:19.122849    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84b5f84777-txqd7"
Oct 30 21:18:19.225193 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:19.225157    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:18:19.225414 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:19.225184    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:18:19.225414 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:19.225245    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rswnw\""
Oct 30 21:18:19.225414 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:19.225268    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 21:18:19.226104 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:19.226083    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 21:18:19.230117 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:19.230091    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84b5f84777-txqd7"]
Oct 30 21:18:19.282541 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:19.282508    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62359a1a-6a7c-482a-8269-9a1548cb02ee-metrics-certs\") pod \"maas-api-84b5f84777-txqd7\" (UID: \"62359a1a-6a7c-482a-8269-9a1548cb02ee\") " pod="odh-ai-gateway-infra/maas-api-84b5f84777-txqd7"
Oct 30 21:18:19.282716 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:19.282545    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62359a1a-6a7c-482a-8269-9a1548cb02ee-maas-api-tls\") pod \"maas-api-84b5f84777-txqd7\" (UID: \"62359a1a-6a7c-482a-8269-9a1548cb02ee\") " pod="odh-ai-gateway-infra/maas-api-84b5f84777-txqd7"
Oct 30 21:18:19.282716 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:19.282572    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62359a1a-6a7c-482a-8269-9a1548cb02ee-openshift-service-ca\") pod \"maas-api-84b5f84777-txqd7\" (UID: \"62359a1a-6a7c-482a-8269-9a1548cb02ee\") " pod="odh-ai-gateway-infra/maas-api-84b5f84777-txqd7"
Oct 30 21:18:19.282813 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:19.282708    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-657m9\" (UniqueName: \"kubernetes.io/projected/62359a1a-6a7c-482a-8269-9a1548cb02ee-kube-api-access-657m9\") pod \"maas-api-84b5f84777-txqd7\" (UID: \"62359a1a-6a7c-482a-8269-9a1548cb02ee\") " pod="odh-ai-gateway-infra/maas-api-84b5f84777-txqd7"
Oct 30 21:18:19.384004 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:19.383919    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62359a1a-6a7c-482a-8269-9a1548cb02ee-metrics-certs\") pod \"maas-api-84b5f84777-txqd7\" (UID: \"62359a1a-6a7c-482a-8269-9a1548cb02ee\") " pod="odh-ai-gateway-infra/maas-api-84b5f84777-txqd7"
Oct 30 21:18:19.384004 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:19.383957    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62359a1a-6a7c-482a-8269-9a1548cb02ee-maas-api-tls\") pod \"maas-api-84b5f84777-txqd7\" (UID: \"62359a1a-6a7c-482a-8269-9a1548cb02ee\") " pod="odh-ai-gateway-infra/maas-api-84b5f84777-txqd7"
Oct 30 21:18:19.384004 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:19.383983    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62359a1a-6a7c-482a-8269-9a1548cb02ee-openshift-service-ca\") pod \"maas-api-84b5f84777-txqd7\" (UID: \"62359a1a-6a7c-482a-8269-9a1548cb02ee\") " pod="odh-ai-gateway-infra/maas-api-84b5f84777-txqd7"
Oct 30 21:18:19.384308 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:19.384010    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-657m9\" (UniqueName: \"kubernetes.io/projected/62359a1a-6a7c-482a-8269-9a1548cb02ee-kube-api-access-657m9\") pod \"maas-api-84b5f84777-txqd7\" (UID: \"62359a1a-6a7c-482a-8269-9a1548cb02ee\") " pod="odh-ai-gateway-infra/maas-api-84b5f84777-txqd7"
Oct 30 21:18:19.384680 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:19.384652    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62359a1a-6a7c-482a-8269-9a1548cb02ee-openshift-service-ca\") pod \"maas-api-84b5f84777-txqd7\" (UID: \"62359a1a-6a7c-482a-8269-9a1548cb02ee\") " pod="odh-ai-gateway-infra/maas-api-84b5f84777-txqd7"
Oct 30 21:18:19.386738 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:19.386716    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62359a1a-6a7c-482a-8269-9a1548cb02ee-metrics-certs\") pod \"maas-api-84b5f84777-txqd7\" (UID: \"62359a1a-6a7c-482a-8269-9a1548cb02ee\") " pod="odh-ai-gateway-infra/maas-api-84b5f84777-txqd7"
Oct 30 21:18:19.386927 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:19.386863    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62359a1a-6a7c-482a-8269-9a1548cb02ee-maas-api-tls\") pod \"maas-api-84b5f84777-txqd7\" (UID: \"62359a1a-6a7c-482a-8269-9a1548cb02ee\") " pod="odh-ai-gateway-infra/maas-api-84b5f84777-txqd7"
Oct 30 21:18:19.442107 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:19.442072    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-657m9\" (UniqueName: \"kubernetes.io/projected/62359a1a-6a7c-482a-8269-9a1548cb02ee-kube-api-access-657m9\") pod \"maas-api-84b5f84777-txqd7\" (UID: \"62359a1a-6a7c-482a-8269-9a1548cb02ee\") " pod="odh-ai-gateway-infra/maas-api-84b5f84777-txqd7"
Oct 30 21:18:19.735846 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:19.735735    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84b5f84777-txqd7"
Oct 30 21:18:20.109991 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:20.109966    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84b5f84777-txqd7"]
Oct 30 21:18:20.111851 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:18:20.111805    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62359a1a_6a7c_482a_8269_9a1548cb02ee.slice/crio-099dfcea37f9036d53b51e58cca53957210c3ca8ec7979d3e57514856c8dac45 WatchSource:0}: Error finding container 099dfcea37f9036d53b51e58cca53957210c3ca8ec7979d3e57514856c8dac45: Status 404 returned error can't find the container with id 099dfcea37f9036d53b51e58cca53957210c3ca8ec7979d3e57514856c8dac45
Oct 30 21:18:20.113164 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:20.113145    5071 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:18:20.406609 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:20.406530    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84b5f84777-txqd7" event={"ID":"62359a1a-6a7c-482a-8269-9a1548cb02ee","Type":"ContainerStarted","Data":"099dfcea37f9036d53b51e58cca53957210c3ca8ec7979d3e57514856c8dac45"}
Oct 30 21:18:21.413031 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:21.412992    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84b5f84777-txqd7" event={"ID":"62359a1a-6a7c-482a-8269-9a1548cb02ee","Type":"ContainerStarted","Data":"e43ede00428638e23ea396081bb71dfb94109ca48bae09c0771d89196e94a532"}
Oct 30 21:18:21.413461 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:21.413121    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-84b5f84777-txqd7"
Oct 30 21:18:21.529886 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:21.529799    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-84b5f84777-txqd7" podStartSLOduration=1.688809974 podStartE2EDuration="2.529785785s" totalImagesPullingTime="840.975811ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:21.527946847 +0000 UTC m=+3113.444968395" watchObservedRunningTime="2026-10-30 21:18:21.529785785 +0000 UTC m=+3113.446807398"
Oct 30 21:18:25.835308 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:25.835267    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:18:26.533595 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:26.533554    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:18:26.725426 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:26.725390    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:18:26.974920 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:26.974886    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-84b5f84777-txqd7"]
Oct 30 21:18:26.975352 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:26.975151    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-84b5f84777-txqd7" podUID="62359a1a-6a7c-482a-8269-9a1548cb02ee" containerName="maas-api" containerID="cri-o://e43ede00428638e23ea396081bb71dfb94109ca48bae09c0771d89196e94a532" gracePeriod=30
Oct 30 21:18:26.981277 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:26.981255    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-84b5f84777-txqd7"
Oct 30 21:18:27.023478 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.023448    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d9876d865-9rcvl"]
Oct 30 21:18:27.027328 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.027308    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d9876d865-9rcvl"
Oct 30 21:18:27.065918 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.065885    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfp49\" (UniqueName: \"kubernetes.io/projected/d626d86a-5d08-4929-af30-3d124bce8f7d-kube-api-access-cfp49\") pod \"maas-api-6d9876d865-9rcvl\" (UID: \"d626d86a-5d08-4929-af30-3d124bce8f7d\") " pod="odh-ai-gateway-infra/maas-api-6d9876d865-9rcvl"
Oct 30 21:18:27.066097 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.065933    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d626d86a-5d08-4929-af30-3d124bce8f7d-metrics-certs\") pod \"maas-api-6d9876d865-9rcvl\" (UID: \"d626d86a-5d08-4929-af30-3d124bce8f7d\") " pod="odh-ai-gateway-infra/maas-api-6d9876d865-9rcvl"
Oct 30 21:18:27.066097 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.066052    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d626d86a-5d08-4929-af30-3d124bce8f7d-openshift-service-ca\") pod \"maas-api-6d9876d865-9rcvl\" (UID: \"d626d86a-5d08-4929-af30-3d124bce8f7d\") " pod="odh-ai-gateway-infra/maas-api-6d9876d865-9rcvl"
Oct 30 21:18:27.066200 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.066114    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d626d86a-5d08-4929-af30-3d124bce8f7d-maas-api-tls\") pod \"maas-api-6d9876d865-9rcvl\" (UID: \"d626d86a-5d08-4929-af30-3d124bce8f7d\") " pod="odh-ai-gateway-infra/maas-api-6d9876d865-9rcvl"
Oct 30 21:18:27.079220 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.079183    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d9876d865-9rcvl"]
Oct 30 21:18:27.172039 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.166795    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfp49\" (UniqueName: \"kubernetes.io/projected/d626d86a-5d08-4929-af30-3d124bce8f7d-kube-api-access-cfp49\") pod \"maas-api-6d9876d865-9rcvl\" (UID: \"d626d86a-5d08-4929-af30-3d124bce8f7d\") " pod="odh-ai-gateway-infra/maas-api-6d9876d865-9rcvl"
Oct 30 21:18:27.172039 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.166924    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d626d86a-5d08-4929-af30-3d124bce8f7d-metrics-certs\") pod \"maas-api-6d9876d865-9rcvl\" (UID: \"d626d86a-5d08-4929-af30-3d124bce8f7d\") " pod="odh-ai-gateway-infra/maas-api-6d9876d865-9rcvl"
Oct 30 21:18:27.172039 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.167009    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d626d86a-5d08-4929-af30-3d124bce8f7d-openshift-service-ca\") pod \"maas-api-6d9876d865-9rcvl\" (UID: \"d626d86a-5d08-4929-af30-3d124bce8f7d\") " pod="odh-ai-gateway-infra/maas-api-6d9876d865-9rcvl"
Oct 30 21:18:27.172039 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.167067    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d626d86a-5d08-4929-af30-3d124bce8f7d-maas-api-tls\") pod \"maas-api-6d9876d865-9rcvl\" (UID: \"d626d86a-5d08-4929-af30-3d124bce8f7d\") " pod="odh-ai-gateway-infra/maas-api-6d9876d865-9rcvl"
Oct 30 21:18:27.172039 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.169001    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d626d86a-5d08-4929-af30-3d124bce8f7d-openshift-service-ca\") pod \"maas-api-6d9876d865-9rcvl\" (UID: \"d626d86a-5d08-4929-af30-3d124bce8f7d\") " pod="odh-ai-gateway-infra/maas-api-6d9876d865-9rcvl"
Oct 30 21:18:27.172039 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.170533    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d626d86a-5d08-4929-af30-3d124bce8f7d-metrics-certs\") pod \"maas-api-6d9876d865-9rcvl\" (UID: \"d626d86a-5d08-4929-af30-3d124bce8f7d\") " pod="odh-ai-gateway-infra/maas-api-6d9876d865-9rcvl"
Oct 30 21:18:27.172039 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.171955    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d626d86a-5d08-4929-af30-3d124bce8f7d-maas-api-tls\") pod \"maas-api-6d9876d865-9rcvl\" (UID: \"d626d86a-5d08-4929-af30-3d124bce8f7d\") " pod="odh-ai-gateway-infra/maas-api-6d9876d865-9rcvl"
Oct 30 21:18:27.226460 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.226399    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfp49\" (UniqueName: \"kubernetes.io/projected/d626d86a-5d08-4929-af30-3d124bce8f7d-kube-api-access-cfp49\") pod \"maas-api-6d9876d865-9rcvl\" (UID: \"d626d86a-5d08-4929-af30-3d124bce8f7d\") " pod="odh-ai-gateway-infra/maas-api-6d9876d865-9rcvl"
Oct 30 21:18:27.246742 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.246713    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84b5f84777-txqd7"
Oct 30 21:18:27.267844 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.267800    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62359a1a-6a7c-482a-8269-9a1548cb02ee-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62359a1a-6a7c-482a-8269-9a1548cb02ee-openshift-service-ca\") pod \"62359a1a-6a7c-482a-8269-9a1548cb02ee\" (UID: \"62359a1a-6a7c-482a-8269-9a1548cb02ee\") "
Oct 30 21:18:27.267968 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.267868    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62359a1a-6a7c-482a-8269-9a1548cb02ee-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62359a1a-6a7c-482a-8269-9a1548cb02ee-maas-api-tls\") pod \"62359a1a-6a7c-482a-8269-9a1548cb02ee\" (UID: \"62359a1a-6a7c-482a-8269-9a1548cb02ee\") "
Oct 30 21:18:27.267968 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.267917    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62359a1a-6a7c-482a-8269-9a1548cb02ee-kube-api-access-657m9\" (UniqueName: \"kubernetes.io/projected/62359a1a-6a7c-482a-8269-9a1548cb02ee-kube-api-access-657m9\") pod \"62359a1a-6a7c-482a-8269-9a1548cb02ee\" (UID: \"62359a1a-6a7c-482a-8269-9a1548cb02ee\") "
Oct 30 21:18:27.267968 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.267951    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62359a1a-6a7c-482a-8269-9a1548cb02ee-metrics-certs\" (UniqueName: \"kubernetes.io/secret/62359a1a-6a7c-482a-8269-9a1548cb02ee-metrics-certs\") pod \"62359a1a-6a7c-482a-8269-9a1548cb02ee\" (UID: \"62359a1a-6a7c-482a-8269-9a1548cb02ee\") "
Oct 30 21:18:27.268163 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.268140    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62359a1a-6a7c-482a-8269-9a1548cb02ee-openshift-service-ca" pod "62359a1a-6a7c-482a-8269-9a1548cb02ee" (UID: "62359a1a-6a7c-482a-8269-9a1548cb02ee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:27.268415 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.268395    5071 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62359a1a-6a7c-482a-8269-9a1548cb02ee-openshift-service-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:27.270024 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.270000    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62359a1a-6a7c-482a-8269-9a1548cb02ee-maas-api-tls" pod "62359a1a-6a7c-482a-8269-9a1548cb02ee" (UID: "62359a1a-6a7c-482a-8269-9a1548cb02ee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:18:27.270113 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.270051    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62359a1a-6a7c-482a-8269-9a1548cb02ee-metrics-certs" pod "62359a1a-6a7c-482a-8269-9a1548cb02ee" (UID: "62359a1a-6a7c-482a-8269-9a1548cb02ee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:18:27.270113 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.270093    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62359a1a-6a7c-482a-8269-9a1548cb02ee-kube-api-access-657m9" pod "62359a1a-6a7c-482a-8269-9a1548cb02ee" (UID: "62359a1a-6a7c-482a-8269-9a1548cb02ee"). InnerVolumeSpecName "kube-api-access-657m9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:27.368906 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.368867    5071 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62359a1a-6a7c-482a-8269-9a1548cb02ee-metrics-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:27.368906 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.368897    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62359a1a-6a7c-482a-8269-9a1548cb02ee-maas-api-tls\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:27.368906 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.368908    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-657m9\" (UniqueName: \"kubernetes.io/projected/62359a1a-6a7c-482a-8269-9a1548cb02ee-kube-api-access-657m9\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:27.386571 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.386541    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d9876d865-9rcvl"
Oct 30 21:18:27.459591 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.456594    5071 generic.go:356] "Generic (PLEG): container finished" podID="62359a1a-6a7c-482a-8269-9a1548cb02ee" containerID="e43ede00428638e23ea396081bb71dfb94109ca48bae09c0771d89196e94a532" exitCode=0
Oct 30 21:18:27.459591 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.456711    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84b5f84777-txqd7" event={"ID":"62359a1a-6a7c-482a-8269-9a1548cb02ee","Type":"ContainerDied","Data":"e43ede00428638e23ea396081bb71dfb94109ca48bae09c0771d89196e94a532"}
Oct 30 21:18:27.459591 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.456761    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84b5f84777-txqd7" event={"ID":"62359a1a-6a7c-482a-8269-9a1548cb02ee","Type":"ContainerDied","Data":"099dfcea37f9036d53b51e58cca53957210c3ca8ec7979d3e57514856c8dac45"}
Oct 30 21:18:27.459591 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.456790    5071 scope.go:122] "RemoveContainer" containerID="e43ede00428638e23ea396081bb71dfb94109ca48bae09c0771d89196e94a532"
Oct 30 21:18:27.459591 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.456999    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84b5f84777-txqd7"
Oct 30 21:18:27.472673 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.472649    5071 scope.go:122] "RemoveContainer" containerID="e43ede00428638e23ea396081bb71dfb94109ca48bae09c0771d89196e94a532"
Oct 30 21:18:27.473084 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:18:27.473018    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e43ede00428638e23ea396081bb71dfb94109ca48bae09c0771d89196e94a532\": container with ID starting with e43ede00428638e23ea396081bb71dfb94109ca48bae09c0771d89196e94a532 not found: ID does not exist" containerID="e43ede00428638e23ea396081bb71dfb94109ca48bae09c0771d89196e94a532"
Oct 30 21:18:27.473084 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.473050    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e43ede00428638e23ea396081bb71dfb94109ca48bae09c0771d89196e94a532"} err="failed to get container status \"e43ede00428638e23ea396081bb71dfb94109ca48bae09c0771d89196e94a532\": rpc error: code = NotFound desc = could not find container \"e43ede00428638e23ea396081bb71dfb94109ca48bae09c0771d89196e94a532\": container with ID starting with e43ede00428638e23ea396081bb71dfb94109ca48bae09c0771d89196e94a532 not found: ID does not exist"
Oct 30 21:18:27.624447 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.624413    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-84b5f84777-txqd7"]
Oct 30 21:18:27.627565 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.627542    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-84b5f84777-txqd7"]
Oct 30 21:18:27.764371 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:27.764348    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d9876d865-9rcvl"]
Oct 30 21:18:27.766481 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:18:27.766446    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd626d86a_5d08_4929_af30_3d124bce8f7d.slice/crio-e0c76c7f6647a6ea5be8b69c63a9ae0697e6a8f7253fa7d8adc2a2da7b5a6558 WatchSource:0}: Error finding container e0c76c7f6647a6ea5be8b69c63a9ae0697e6a8f7253fa7d8adc2a2da7b5a6558: Status 404 returned error can't find the container with id e0c76c7f6647a6ea5be8b69c63a9ae0697e6a8f7253fa7d8adc2a2da7b5a6558
Oct 30 21:18:28.200363 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:28.200330    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62359a1a-6a7c-482a-8269-9a1548cb02ee" path="/var/lib/kubelet/pods/62359a1a-6a7c-482a-8269-9a1548cb02ee/volumes"
Oct 30 21:18:28.462773 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:28.462734    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d9876d865-9rcvl" event={"ID":"d626d86a-5d08-4929-af30-3d124bce8f7d","Type":"ContainerStarted","Data":"e0c76c7f6647a6ea5be8b69c63a9ae0697e6a8f7253fa7d8adc2a2da7b5a6558"}
Oct 30 21:18:29.471422 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:29.471389    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d9876d865-9rcvl" event={"ID":"d626d86a-5d08-4929-af30-3d124bce8f7d","Type":"ContainerStarted","Data":"973de3d23be1bc747cb6eec7dcbea31e101d89c6be9d3700a3955f7ad19d3303"}
Oct 30 21:18:29.471858 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:29.471574    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d9876d865-9rcvl"
Oct 30 21:18:29.587173 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:29.587114    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d9876d865-9rcvl" podStartSLOduration=2.70056041 podStartE2EDuration="3.587098409s" totalImagesPullingTime="886.537999ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:29.585978145 +0000 UTC m=+3121.502999681" watchObservedRunningTime="2026-10-30 21:18:29.587098409 +0000 UTC m=+3121.504119946"
Oct 30 21:18:30.051615 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:30.051585    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmh6k"]
Oct 30 21:18:30.055352 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:30.055317    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmh6k"]
Oct 30 21:18:30.112239 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:30.112202    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:18:30.201409 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:30.201370    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbbadd11-9ab2-4424-a66a-82f3d29423f9" path="/var/lib/kubelet/pods/bbbadd11-9ab2-4424-a66a-82f3d29423f9/volumes"
Oct 30 21:18:30.288187 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:30.288151    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:18:30.459878 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:30.459758    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:18:33.407177 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:33.407137    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:18:33.708867 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:33.708758    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:18:33.831413 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:33.831373    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bddeea9c-bd7c96b99-8dktz"]
Oct 30 21:18:33.832028 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:33.832011    5071 cpu_manager.go:414] "containerMap: removing container" podUID="62359a1a-6a7c-482a-8269-9a1548cb02ee" containerName="maas-api"
Oct 30 21:18:33.832092 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:33.832030    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62359a1a-6a7c-482a-8269-9a1548cb02ee" containerName="maas-api"
Oct 30 21:18:33.832140 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:33.832132    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="62359a1a-6a7c-482a-8269-9a1548cb02ee" containerName="maas-api"
Oct 30 21:18:33.835499 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:33.835483    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bddeea9c-bd7c96b99-8dktz"
Oct 30 21:18:33.932506 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:33.932475    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxgc9\" (UniqueName: \"kubernetes.io/projected/74cd503a-e270-489a-8876-6b27084a9981-kube-api-access-dxgc9\") pod \"payload-pre-processing-e2e-ait-bddeea9c-bd7c96b99-8dktz\" (UID: \"74cd503a-e270-489a-8876-6b27084a9981\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bddeea9c-bd7c96b99-8dktz"
Oct 30 21:18:33.932736 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:33.932592    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74cd503a-e270-489a-8876-6b27084a9981-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bddeea9c-bd7c96b99-8dktz\" (UID: \"74cd503a-e270-489a-8876-6b27084a9981\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bddeea9c-bd7c96b99-8dktz"
Oct 30 21:18:33.943882 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:33.943859    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-bddeea9c\""
Oct 30 21:18:33.944165 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:33.944146    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-bddeea9c-dockercfg-l2mnk\""
Oct 30 21:18:33.947904 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:33.947881    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bddeea9c-bd7c96b99-8dktz"]
Oct 30 21:18:33.996691 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:33.996616    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bddeea9c-f85cdfbdd-nr9mz"]
Oct 30 21:18:34.000419 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:34.000401    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bddeea9c-f85cdfbdd-nr9mz"
Oct 30 21:18:34.033419 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:34.033387    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74cd503a-e270-489a-8876-6b27084a9981-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bddeea9c-bd7c96b99-8dktz\" (UID: \"74cd503a-e270-489a-8876-6b27084a9981\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bddeea9c-bd7c96b99-8dktz"
Oct 30 21:18:34.033590 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:34.033477    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g76ns\" (UniqueName: \"kubernetes.io/projected/73bbb8ac-402c-4717-bc58-a8a9c5d95ab8-kube-api-access-g76ns\") pod \"payload-processing-e2e-ait-bddeea9c-f85cdfbdd-nr9mz\" (UID: \"73bbb8ac-402c-4717-bc58-a8a9c5d95ab8\") " pod="openshift-ingress/payload-processing-e2e-ait-bddeea9c-f85cdfbdd-nr9mz"
Oct 30 21:18:34.033590 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:34.033528    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxgc9\" (UniqueName: \"kubernetes.io/projected/74cd503a-e270-489a-8876-6b27084a9981-kube-api-access-dxgc9\") pod \"payload-pre-processing-e2e-ait-bddeea9c-bd7c96b99-8dktz\" (UID: \"74cd503a-e270-489a-8876-6b27084a9981\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bddeea9c-bd7c96b99-8dktz"
Oct 30 21:18:34.033590 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:34.033576    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73bbb8ac-402c-4717-bc58-a8a9c5d95ab8-plugins-config-volume\") pod \"payload-processing-e2e-ait-bddeea9c-f85cdfbdd-nr9mz\" (UID: \"73bbb8ac-402c-4717-bc58-a8a9c5d95ab8\") " pod="openshift-ingress/payload-processing-e2e-ait-bddeea9c-f85cdfbdd-nr9mz"
Oct 30 21:18:34.034022 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:34.033997    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bddeea9c-f85cdfbdd-nr9mz"]
Oct 30 21:18:34.034276 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:34.034253    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74cd503a-e270-489a-8876-6b27084a9981-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bddeea9c-bd7c96b99-8dktz\" (UID: \"74cd503a-e270-489a-8876-6b27084a9981\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bddeea9c-bd7c96b99-8dktz"
Oct 30 21:18:34.092360 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:34.092321    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxgc9\" (UniqueName: \"kubernetes.io/projected/74cd503a-e270-489a-8876-6b27084a9981-kube-api-access-dxgc9\") pod \"payload-pre-processing-e2e-ait-bddeea9c-bd7c96b99-8dktz\" (UID: \"74cd503a-e270-489a-8876-6b27084a9981\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bddeea9c-bd7c96b99-8dktz"
Oct 30 21:18:34.134976 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:34.134933    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g76ns\" (UniqueName: \"kubernetes.io/projected/73bbb8ac-402c-4717-bc58-a8a9c5d95ab8-kube-api-access-g76ns\") pod \"payload-processing-e2e-ait-bddeea9c-f85cdfbdd-nr9mz\" (UID: \"73bbb8ac-402c-4717-bc58-a8a9c5d95ab8\") " pod="openshift-ingress/payload-processing-e2e-ait-bddeea9c-f85cdfbdd-nr9mz"
Oct 30 21:18:34.135160 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:34.135050    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73bbb8ac-402c-4717-bc58-a8a9c5d95ab8-plugins-config-volume\") pod \"payload-processing-e2e-ait-bddeea9c-f85cdfbdd-nr9mz\" (UID: \"73bbb8ac-402c-4717-bc58-a8a9c5d95ab8\") " pod="openshift-ingress/payload-processing-e2e-ait-bddeea9c-f85cdfbdd-nr9mz"
Oct 30 21:18:34.136038 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:34.136001    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73bbb8ac-402c-4717-bc58-a8a9c5d95ab8-plugins-config-volume\") pod \"payload-processing-e2e-ait-bddeea9c-f85cdfbdd-nr9mz\" (UID: \"73bbb8ac-402c-4717-bc58-a8a9c5d95ab8\") " pod="openshift-ingress/payload-processing-e2e-ait-bddeea9c-f85cdfbdd-nr9mz"
Oct 30 21:18:34.148739 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:34.148715    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bddeea9c-bd7c96b99-8dktz"
Oct 30 21:18:34.194555 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:34.194511    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g76ns\" (UniqueName: \"kubernetes.io/projected/73bbb8ac-402c-4717-bc58-a8a9c5d95ab8-kube-api-access-g76ns\") pod \"payload-processing-e2e-ait-bddeea9c-f85cdfbdd-nr9mz\" (UID: \"73bbb8ac-402c-4717-bc58-a8a9c5d95ab8\") " pod="openshift-ingress/payload-processing-e2e-ait-bddeea9c-f85cdfbdd-nr9mz"
Oct 30 21:18:34.313671 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:34.313639    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bddeea9c-f85cdfbdd-nr9mz"
Oct 30 21:18:34.401593 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:34.400855    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q"]
Oct 30 21:18:34.406281 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:34.406254    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q"
Oct 30 21:18:34.439437 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:34.439396    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afde9863-e123-436a-bf96-80dfd6cc7664-metrics-certs\") pod \"maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q\" (UID: \"afde9863-e123-436a-bf96-80dfd6cc7664\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q"
Oct 30 21:18:34.439790 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:34.439530    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79skq\" (UniqueName: \"kubernetes.io/projected/afde9863-e123-436a-bf96-80dfd6cc7664-kube-api-access-79skq\") pod \"maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q\" (UID: \"afde9863-e123-436a-bf96-80dfd6cc7664\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q"
Oct 30 21:18:34.439790 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:34.439583    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afde9863-e123-436a-bf96-80dfd6cc7664-maas-api-tls\") pod \"maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q\" (UID: \"afde9863-e123-436a-bf96-80dfd6cc7664\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q"
Oct 30 21:18:34.439790 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:34.439656    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afde9863-e123-436a-bf96-80dfd6cc7664-openshift-service-ca\") pod \"maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q\" (UID: \"afde9863-e123-436a-bf96-80dfd6cc7664\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q"
Oct 30 21:18:34.464539 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:34.461030    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-bddeea9c-serving-cert\""
Oct 30 21:18:34.466262 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:34.466237    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q"]
Oct 30 21:18:34.540875 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:34.540815    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afde9863-e123-436a-bf96-80dfd6cc7664-metrics-certs\") pod \"maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q\" (UID: \"afde9863-e123-436a-bf96-80dfd6cc7664\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q"
Oct 30 21:18:34.541065 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:34.540940    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79skq\" (UniqueName: \"kubernetes.io/projected/afde9863-e123-436a-bf96-80dfd6cc7664-kube-api-access-79skq\") pod \"maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q\" (UID: \"afde9863-e123-436a-bf96-80dfd6cc7664\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q"
Oct 30 21:18:34.541065 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:34.540973    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afde9863-e123-436a-bf96-80dfd6cc7664-maas-api-tls\") pod \"maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q\" (UID: \"afde9863-e123-436a-bf96-80dfd6cc7664\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q"
Oct 30 21:18:34.541065 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:34.541010    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afde9863-e123-436a-bf96-80dfd6cc7664-openshift-service-ca\") pod \"maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q\" (UID: \"afde9863-e123-436a-bf96-80dfd6cc7664\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q"
Oct 30 21:18:34.541250 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:18:34.541112    5071 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-bddeea9c-serving-cert: secret "maas-api-e2e-ait-bddeea9c-serving-cert" not found
Oct 30 21:18:34.541250 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:18:34.541179    5071 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afde9863-e123-436a-bf96-80dfd6cc7664-maas-api-tls podName:afde9863-e123-436a-bf96-80dfd6cc7664 nodeName:}" failed. No retries permitted until 2026-10-30 21:18:35.041156341 +0000 UTC m=+3126.958177871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/afde9863-e123-436a-bf96-80dfd6cc7664-maas-api-tls") pod "maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q" (UID: "afde9863-e123-436a-bf96-80dfd6cc7664") : secret "maas-api-e2e-ait-bddeea9c-serving-cert" not found
Oct 30 21:18:34.541810 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:34.541788    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afde9863-e123-436a-bf96-80dfd6cc7664-openshift-service-ca\") pod \"maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q\" (UID: \"afde9863-e123-436a-bf96-80dfd6cc7664\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q"
Oct 30 21:18:34.543447 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:34.543427    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afde9863-e123-436a-bf96-80dfd6cc7664-metrics-certs\") pod \"maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q\" (UID: \"afde9863-e123-436a-bf96-80dfd6cc7664\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q"
Oct 30 21:18:34.573492 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:34.573463    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bddeea9c-bd7c96b99-8dktz"]
Oct 30 21:18:34.574672 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:18:34.574645    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74cd503a_e270_489a_8876_6b27084a9981.slice/crio-ec68aa216b5964ab60645a9e55c43f26027088e3ef646a6d82b0f950560bdb25 WatchSource:0}: Error finding container ec68aa216b5964ab60645a9e55c43f26027088e3ef646a6d82b0f950560bdb25: Status 404 returned error can't find the container with id ec68aa216b5964ab60645a9e55c43f26027088e3ef646a6d82b0f950560bdb25
Oct 30 21:18:34.600321 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:34.600296    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79skq\" (UniqueName: \"kubernetes.io/projected/afde9863-e123-436a-bf96-80dfd6cc7664-kube-api-access-79skq\") pod \"maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q\" (UID: \"afde9863-e123-436a-bf96-80dfd6cc7664\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q"
Oct 30 21:18:34.623013 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:34.622964    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:18:34.692106 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:34.692083    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bddeea9c-f85cdfbdd-nr9mz"]
Oct 30 21:18:34.693635 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:18:34.693606    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73bbb8ac_402c_4717_bc58_a8a9c5d95ab8.slice/crio-ff6ae51c79ffade493745feee7351ef1039c165c830bc65573263620bbafa610 WatchSource:0}: Error finding container ff6ae51c79ffade493745feee7351ef1039c165c830bc65573263620bbafa610: Status 404 returned error can't find the container with id ff6ae51c79ffade493745feee7351ef1039c165c830bc65573263620bbafa610
Oct 30 21:18:34.786175 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:34.786145    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:18:35.046441 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:35.046409    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afde9863-e123-436a-bf96-80dfd6cc7664-maas-api-tls\") pod \"maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q\" (UID: \"afde9863-e123-436a-bf96-80dfd6cc7664\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q"
Oct 30 21:18:35.049032 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:35.049012    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afde9863-e123-436a-bf96-80dfd6cc7664-maas-api-tls\") pod \"maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q\" (UID: \"afde9863-e123-436a-bf96-80dfd6cc7664\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q"
Oct 30 21:18:35.323730 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:35.323659    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q"
Oct 30 21:18:35.481889 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:35.481858    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d9876d865-9rcvl"
Oct 30 21:18:35.515721 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:35.515684    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bddeea9c-bd7c96b99-8dktz" event={"ID":"74cd503a-e270-489a-8876-6b27084a9981","Type":"ContainerStarted","Data":"3de9af5da7ebe6c1f69e1f8a912d9c1ee194b92d2ea84ac01abf5134de0868bb"}
Oct 30 21:18:35.515982 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:35.515747    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-bddeea9c-bd7c96b99-8dktz"
Oct 30 21:18:35.515982 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:35.515770    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bddeea9c-bd7c96b99-8dktz" event={"ID":"74cd503a-e270-489a-8876-6b27084a9981","Type":"ContainerStarted","Data":"ec68aa216b5964ab60645a9e55c43f26027088e3ef646a6d82b0f950560bdb25"}
Oct 30 21:18:35.517362 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:35.517339    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bddeea9c-f85cdfbdd-nr9mz" event={"ID":"73bbb8ac-402c-4717-bc58-a8a9c5d95ab8","Type":"ContainerStarted","Data":"4e71662c109a5bb1e6fead875a8b9d6315d96163cc8e0487e365d91cd7562fbf"}
Oct 30 21:18:35.517454 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:35.517382    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bddeea9c-f85cdfbdd-nr9mz" event={"ID":"73bbb8ac-402c-4717-bc58-a8a9c5d95ab8","Type":"ContainerStarted","Data":"ff6ae51c79ffade493745feee7351ef1039c165c830bc65573263620bbafa610"}
Oct 30 21:18:35.517454 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:35.517403    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-bddeea9c-f85cdfbdd-nr9mz"
Oct 30 21:18:35.716391 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:35.716294    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-bddeea9c-bd7c96b99-8dktz" podStartSLOduration=2.716279203 podStartE2EDuration="2.716279203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:35.714549913 +0000 UTC m=+3127.631571448" watchObservedRunningTime="2026-10-30 21:18:35.716279203 +0000 UTC m=+3127.633300739"
Oct 30 21:18:35.753856 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:35.753812    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q"]
Oct 30 21:18:35.755199 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:18:35.755172    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafde9863_e123_436a_bf96_80dfd6cc7664.slice/crio-b088ee6c10bd4333e0cc6f8f4ff42c7474318034f28358b87efee59587c5d012 WatchSource:0}: Error finding container b088ee6c10bd4333e0cc6f8f4ff42c7474318034f28358b87efee59587c5d012: Status 404 returned error can't find the container with id b088ee6c10bd4333e0cc6f8f4ff42c7474318034f28358b87efee59587c5d012
Oct 30 21:18:35.849764 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:35.849706    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-bddeea9c-f85cdfbdd-nr9mz" podStartSLOduration=2.849690337 podStartE2EDuration="2.849690337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:35.848399837 +0000 UTC m=+3127.765421373" watchObservedRunningTime="2026-10-30 21:18:35.849690337 +0000 UTC m=+3127.766711864"
Oct 30 21:18:36.523964 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:36.523924    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q" event={"ID":"afde9863-e123-436a-bf96-80dfd6cc7664","Type":"ContainerStarted","Data":"b088ee6c10bd4333e0cc6f8f4ff42c7474318034f28358b87efee59587c5d012"}
Oct 30 21:18:37.531312 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:37.531278    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q" event={"ID":"afde9863-e123-436a-bf96-80dfd6cc7664","Type":"ContainerStarted","Data":"638a6cc3e8e5cab3df3d42ecd46944252674736a7c7ab626fe03372340fb8bfb"}
Oct 30 21:18:37.531711 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:37.531337    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q"
Oct 30 21:18:37.651668 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:37.651615    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q" podStartSLOduration=2.793971249 podStartE2EDuration="3.651601247s" totalImagesPullingTime="857.629998ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:37.649879137 +0000 UTC m=+3129.566900674" watchObservedRunningTime="2026-10-30 21:18:37.651601247 +0000 UTC m=+3129.568622828"
Oct 30 21:18:39.965693 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:39.965648    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:18:42.944696 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:42.944658    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:18:43.524378 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:43.524330    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:18:43.543131 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:43.543104    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q"
Oct 30 21:18:44.975869 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:44.975800    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:18:46.174504 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:46.174467    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:18:46.527660 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:46.527582    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-bddeea9c-f85cdfbdd-nr9mz"
Oct 30 21:18:46.527660 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:46.527657    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-bddeea9c-bd7c96b99-8dktz"
Oct 30 21:18:46.729129 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:18:46.729079    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:18:49.897037 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:49.896996    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:18:55.110759 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:55.110725    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:18:55.725350 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:18:55.725313    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:00.313961 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:00.307162    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:05.570521 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:05.570486    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:10.460770 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:10.460723    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:15.390203 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:15.390160    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:16.727206 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:19:16.727167    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:19:18.602890 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:18.602858    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bddeea9c-f85cdfbdd-nr9mz"]
Oct 30 21:19:18.714085 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:18.714054    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bddeea9c-bd7c96b99-8dktz"]
Oct 30 21:19:18.996425 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:18.996342    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:19.204870 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:19.204832    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:20.362462 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:20.362422    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:23.603602 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:23.603560    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-bddeea9c-f85cdfbdd-nr9mz" podUID="73bbb8ac-402c-4717-bc58-a8a9c5d95ab8" containerName="payload-processing" containerID="cri-o://4e71662c109a5bb1e6fead875a8b9d6315d96163cc8e0487e365d91cd7562fbf" gracePeriod=25
Oct 30 21:19:23.715073 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:23.715028    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-bddeea9c-bd7c96b99-8dktz" podUID="74cd503a-e270-489a-8876-6b27084a9981" containerName="payload-pre-processing" containerID="cri-o://3de9af5da7ebe6c1f69e1f8a912d9c1ee194b92d2ea84ac01abf5134de0868bb" gracePeriod=25
Oct 30 21:19:23.793939 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:23.793903    5071 generic.go:356] "Generic (PLEG): container finished" podID="73bbb8ac-402c-4717-bc58-a8a9c5d95ab8" containerID="4e71662c109a5bb1e6fead875a8b9d6315d96163cc8e0487e365d91cd7562fbf" exitCode=0
Oct 30 21:19:23.794080 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:23.793969    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bddeea9c-f85cdfbdd-nr9mz" event={"ID":"73bbb8ac-402c-4717-bc58-a8a9c5d95ab8","Type":"ContainerDied","Data":"4e71662c109a5bb1e6fead875a8b9d6315d96163cc8e0487e365d91cd7562fbf"}
Oct 30 21:19:23.875094 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:23.875070    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bddeea9c-f85cdfbdd-nr9mz"
Oct 30 21:19:23.955606 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:23.955440    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73bbb8ac-402c-4717-bc58-a8a9c5d95ab8-kube-api-access-g76ns\" (UniqueName: \"kubernetes.io/projected/73bbb8ac-402c-4717-bc58-a8a9c5d95ab8-kube-api-access-g76ns\") pod \"73bbb8ac-402c-4717-bc58-a8a9c5d95ab8\" (UID: \"73bbb8ac-402c-4717-bc58-a8a9c5d95ab8\") "
Oct 30 21:19:23.955606 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:23.955515    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73bbb8ac-402c-4717-bc58-a8a9c5d95ab8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73bbb8ac-402c-4717-bc58-a8a9c5d95ab8-plugins-config-volume\") pod \"73bbb8ac-402c-4717-bc58-a8a9c5d95ab8\" (UID: \"73bbb8ac-402c-4717-bc58-a8a9c5d95ab8\") "
Oct 30 21:19:23.956018 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:23.955993    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73bbb8ac-402c-4717-bc58-a8a9c5d95ab8-plugins-config-volume" pod "73bbb8ac-402c-4717-bc58-a8a9c5d95ab8" (UID: "73bbb8ac-402c-4717-bc58-a8a9c5d95ab8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:19:23.957917 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:23.957890    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73bbb8ac-402c-4717-bc58-a8a9c5d95ab8-kube-api-access-g76ns" pod "73bbb8ac-402c-4717-bc58-a8a9c5d95ab8" (UID: "73bbb8ac-402c-4717-bc58-a8a9c5d95ab8"). InnerVolumeSpecName "kube-api-access-g76ns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:23.976341 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:23.976314    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bddeea9c-bd7c96b99-8dktz"
Oct 30 21:19:24.056624 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:24.056597    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74cd503a-e270-489a-8876-6b27084a9981-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74cd503a-e270-489a-8876-6b27084a9981-plugins-config-volume\") pod \"74cd503a-e270-489a-8876-6b27084a9981\" (UID: \"74cd503a-e270-489a-8876-6b27084a9981\") "
Oct 30 21:19:24.056844 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:24.056688    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74cd503a-e270-489a-8876-6b27084a9981-kube-api-access-dxgc9\" (UniqueName: \"kubernetes.io/projected/74cd503a-e270-489a-8876-6b27084a9981-kube-api-access-dxgc9\") pod \"74cd503a-e270-489a-8876-6b27084a9981\" (UID: \"74cd503a-e270-489a-8876-6b27084a9981\") "
Oct 30 21:19:24.057079 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:24.057052    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74cd503a-e270-489a-8876-6b27084a9981-plugins-config-volume" pod "74cd503a-e270-489a-8876-6b27084a9981" (UID: "74cd503a-e270-489a-8876-6b27084a9981"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:19:24.057159 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:24.057064    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g76ns\" (UniqueName: \"kubernetes.io/projected/73bbb8ac-402c-4717-bc58-a8a9c5d95ab8-kube-api-access-g76ns\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:24.057159 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:24.057111    5071 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73bbb8ac-402c-4717-bc58-a8a9c5d95ab8-plugins-config-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:24.062162 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:24.059371    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74cd503a-e270-489a-8876-6b27084a9981-kube-api-access-dxgc9" pod "74cd503a-e270-489a-8876-6b27084a9981" (UID: "74cd503a-e270-489a-8876-6b27084a9981"). InnerVolumeSpecName "kube-api-access-dxgc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:24.158464 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:24.158382    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxgc9\" (UniqueName: \"kubernetes.io/projected/74cd503a-e270-489a-8876-6b27084a9981-kube-api-access-dxgc9\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:24.158464 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:24.158411    5071 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74cd503a-e270-489a-8876-6b27084a9981-plugins-config-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:24.586212 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:24.586180    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q"]
Oct 30 21:19:24.586469 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:24.586442    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q" podUID="afde9863-e123-436a-bf96-80dfd6cc7664" containerName="maas-api" containerID="cri-o://638a6cc3e8e5cab3df3d42ecd46944252674736a7c7ab626fe03372340fb8bfb" gracePeriod=30
Oct 30 21:19:24.708648 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:24.708605    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:24.839422 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:24.839338    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bddeea9c-f85cdfbdd-nr9mz"
Oct 30 21:19:24.839579 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:24.839332    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bddeea9c-f85cdfbdd-nr9mz" event={"ID":"73bbb8ac-402c-4717-bc58-a8a9c5d95ab8","Type":"ContainerDied","Data":"ff6ae51c79ffade493745feee7351ef1039c165c830bc65573263620bbafa610"}
Oct 30 21:19:24.839579 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:24.839486    5071 scope.go:122] "RemoveContainer" containerID="4e71662c109a5bb1e6fead875a8b9d6315d96163cc8e0487e365d91cd7562fbf"
Oct 30 21:19:24.846104 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:24.846071    5071 generic.go:356] "Generic (PLEG): container finished" podID="afde9863-e123-436a-bf96-80dfd6cc7664" containerID="638a6cc3e8e5cab3df3d42ecd46944252674736a7c7ab626fe03372340fb8bfb" exitCode=0
Oct 30 21:19:24.846252 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:24.846111    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q" event={"ID":"afde9863-e123-436a-bf96-80dfd6cc7664","Type":"ContainerDied","Data":"638a6cc3e8e5cab3df3d42ecd46944252674736a7c7ab626fe03372340fb8bfb"}
Oct 30 21:19:24.848097 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:24.847998    5071 generic.go:356] "Generic (PLEG): container finished" podID="74cd503a-e270-489a-8876-6b27084a9981" containerID="3de9af5da7ebe6c1f69e1f8a912d9c1ee194b92d2ea84ac01abf5134de0868bb" exitCode=0
Oct 30 21:19:24.848097 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:24.848045    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bddeea9c-bd7c96b99-8dktz" event={"ID":"74cd503a-e270-489a-8876-6b27084a9981","Type":"ContainerDied","Data":"3de9af5da7ebe6c1f69e1f8a912d9c1ee194b92d2ea84ac01abf5134de0868bb"}
Oct 30 21:19:24.848097 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:24.848085    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bddeea9c-bd7c96b99-8dktz" event={"ID":"74cd503a-e270-489a-8876-6b27084a9981","Type":"ContainerDied","Data":"ec68aa216b5964ab60645a9e55c43f26027088e3ef646a6d82b0f950560bdb25"}
Oct 30 21:19:24.848570 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:24.848539    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bddeea9c-bd7c96b99-8dktz"
Oct 30 21:19:24.851923 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:24.851904    5071 scope.go:122] "RemoveContainer" containerID="3de9af5da7ebe6c1f69e1f8a912d9c1ee194b92d2ea84ac01abf5134de0868bb"
Oct 30 21:19:24.873803 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:24.873785    5071 scope.go:122] "RemoveContainer" containerID="3de9af5da7ebe6c1f69e1f8a912d9c1ee194b92d2ea84ac01abf5134de0868bb"
Oct 30 21:19:24.874071 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:19:24.874050    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3de9af5da7ebe6c1f69e1f8a912d9c1ee194b92d2ea84ac01abf5134de0868bb\": container with ID starting with 3de9af5da7ebe6c1f69e1f8a912d9c1ee194b92d2ea84ac01abf5134de0868bb not found: ID does not exist" containerID="3de9af5da7ebe6c1f69e1f8a912d9c1ee194b92d2ea84ac01abf5134de0868bb"
Oct 30 21:19:24.874128 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:24.874083    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3de9af5da7ebe6c1f69e1f8a912d9c1ee194b92d2ea84ac01abf5134de0868bb"} err="failed to get container status \"3de9af5da7ebe6c1f69e1f8a912d9c1ee194b92d2ea84ac01abf5134de0868bb\": rpc error: code = NotFound desc = could not find container \"3de9af5da7ebe6c1f69e1f8a912d9c1ee194b92d2ea84ac01abf5134de0868bb\": container with ID starting with 3de9af5da7ebe6c1f69e1f8a912d9c1ee194b92d2ea84ac01abf5134de0868bb not found: ID does not exist"
Oct 30 21:19:24.875849 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:24.875832    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q"
Oct 30 21:19:24.978641 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:24.978571    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afde9863-e123-436a-bf96-80dfd6cc7664-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afde9863-e123-436a-bf96-80dfd6cc7664-openshift-service-ca\") pod \"afde9863-e123-436a-bf96-80dfd6cc7664\" (UID: \"afde9863-e123-436a-bf96-80dfd6cc7664\") "
Oct 30 21:19:24.979282 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:24.979247    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afde9863-e123-436a-bf96-80dfd6cc7664-kube-api-access-79skq\" (UniqueName: \"kubernetes.io/projected/afde9863-e123-436a-bf96-80dfd6cc7664-kube-api-access-79skq\") pod \"afde9863-e123-436a-bf96-80dfd6cc7664\" (UID: \"afde9863-e123-436a-bf96-80dfd6cc7664\") "
Oct 30 21:19:24.979408 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:24.979320    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afde9863-e123-436a-bf96-80dfd6cc7664-metrics-certs\" (UniqueName: \"kubernetes.io/secret/afde9863-e123-436a-bf96-80dfd6cc7664-metrics-certs\") pod \"afde9863-e123-436a-bf96-80dfd6cc7664\" (UID: \"afde9863-e123-436a-bf96-80dfd6cc7664\") "
Oct 30 21:19:24.979408 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:24.979365    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afde9863-e123-436a-bf96-80dfd6cc7664-openshift-service-ca" pod "afde9863-e123-436a-bf96-80dfd6cc7664" (UID: "afde9863-e123-436a-bf96-80dfd6cc7664"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:19:24.979408 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:24.979383    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afde9863-e123-436a-bf96-80dfd6cc7664-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afde9863-e123-436a-bf96-80dfd6cc7664-maas-api-tls\") pod \"afde9863-e123-436a-bf96-80dfd6cc7664\" (UID: \"afde9863-e123-436a-bf96-80dfd6cc7664\") "
Oct 30 21:19:24.979804 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:24.979782    5071 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afde9863-e123-436a-bf96-80dfd6cc7664-openshift-service-ca\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:24.987480 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:24.987448    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afde9863-e123-436a-bf96-80dfd6cc7664-maas-api-tls" pod "afde9863-e123-436a-bf96-80dfd6cc7664" (UID: "afde9863-e123-436a-bf96-80dfd6cc7664"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:19:24.988639 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:24.988602    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afde9863-e123-436a-bf96-80dfd6cc7664-metrics-certs" pod "afde9863-e123-436a-bf96-80dfd6cc7664" (UID: "afde9863-e123-436a-bf96-80dfd6cc7664"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:19:24.993336 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:24.993292    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afde9863-e123-436a-bf96-80dfd6cc7664-kube-api-access-79skq" pod "afde9863-e123-436a-bf96-80dfd6cc7664" (UID: "afde9863-e123-436a-bf96-80dfd6cc7664"). InnerVolumeSpecName "kube-api-access-79skq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:25.007738 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:25.007705    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bddeea9c-bd7c96b99-8dktz"]
Oct 30 21:19:25.011168 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:25.011141    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bddeea9c-bd7c96b99-8dktz"]
Oct 30 21:19:25.081313 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:25.081281    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79skq\" (UniqueName: \"kubernetes.io/projected/afde9863-e123-436a-bf96-80dfd6cc7664-kube-api-access-79skq\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:25.081313 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:25.081307    5071 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afde9863-e123-436a-bf96-80dfd6cc7664-metrics-certs\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:25.081313 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:25.081320    5071 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afde9863-e123-436a-bf96-80dfd6cc7664-maas-api-tls\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:25.121407 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:25.121326    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bddeea9c-f85cdfbdd-nr9mz"]
Oct 30 21:19:25.128010 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:25.127985    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bddeea9c-f85cdfbdd-nr9mz"]
Oct 30 21:19:25.854791 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:25.854761    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q"
Oct 30 21:19:25.855346 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:25.854805    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q" event={"ID":"afde9863-e123-436a-bf96-80dfd6cc7664","Type":"ContainerDied","Data":"b088ee6c10bd4333e0cc6f8f4ff42c7474318034f28358b87efee59587c5d012"}
Oct 30 21:19:25.855346 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:25.854903    5071 scope.go:122] "RemoveContainer" containerID="638a6cc3e8e5cab3df3d42ecd46944252674736a7c7ab626fe03372340fb8bfb"
Oct 30 21:19:26.020734 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:26.020697    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q"]
Oct 30 21:19:26.070548 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:26.070516    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bddeea9c-6d695d6b86-2dh5q"]
Oct 30 21:19:26.201060 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:26.200983    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73bbb8ac-402c-4717-bc58-a8a9c5d95ab8" path="/var/lib/kubelet/pods/73bbb8ac-402c-4717-bc58-a8a9c5d95ab8/volumes"
Oct 30 21:19:26.201376 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:26.201363    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74cd503a-e270-489a-8876-6b27084a9981" path="/var/lib/kubelet/pods/74cd503a-e270-489a-8876-6b27084a9981/volumes"
Oct 30 21:19:26.201760 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:26.201747    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afde9863-e123-436a-bf96-80dfd6cc7664" path="/var/lib/kubelet/pods/afde9863-e123-436a-bf96-80dfd6cc7664/volumes"
Oct 30 21:19:26.248171 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:26.248137    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:26.528706 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:26.528612    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:27.492726 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:27.492690    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:19:30.437693 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:30.437652    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:32.005154 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:32.005111    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:32.754007 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:32.753961    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:33.102667 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:33.102620    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:34.473648 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:34.473616    5071 scope.go:122] "RemoveContainer" containerID="ea848bed7c29d88e949ec28cf6d4461115bd0beb09ffe1ff2fa40709fb1e02b5"
Oct 30 21:19:34.482855 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:34.482816    5071 scope.go:122] "RemoveContainer" containerID="c277f9da25e9d03ce37810ec6db35950a98d68af1027845b785a8f39164036ab"
Oct 30 21:19:37.453238 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:37.453150    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:37.717812 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:37.717726    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:37.888302 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:37.888266    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:40.695340 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:40.695289    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:40.892587 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:40.892548    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:41.092104 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:41.092073    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-545519-676cd8876c-bs7ks"]
Oct 30 21:19:41.092561 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:41.092546    5071 cpu_manager.go:414] "containerMap: removing container" podUID="73bbb8ac-402c-4717-bc58-a8a9c5d95ab8" containerName="payload-processing"
Oct 30 21:19:41.092610 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:41.092563    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73bbb8ac-402c-4717-bc58-a8a9c5d95ab8" containerName="payload-processing"
Oct 30 21:19:41.092610 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:41.092586    5071 cpu_manager.go:414] "containerMap: removing container" podUID="74cd503a-e270-489a-8876-6b27084a9981" containerName="payload-pre-processing"
Oct 30 21:19:41.092610 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:41.092592    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74cd503a-e270-489a-8876-6b27084a9981" containerName="payload-pre-processing"
Oct 30 21:19:41.092717 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:41.092623    5071 cpu_manager.go:414] "containerMap: removing container" podUID="afde9863-e123-436a-bf96-80dfd6cc7664" containerName="maas-api"
Oct 30 21:19:41.092717 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:41.092628    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afde9863-e123-436a-bf96-80dfd6cc7664" containerName="maas-api"
Oct 30 21:19:41.092717 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:41.092698    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="afde9863-e123-436a-bf96-80dfd6cc7664" containerName="maas-api"
Oct 30 21:19:41.092717 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:41.092709    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="73bbb8ac-402c-4717-bc58-a8a9c5d95ab8" containerName="payload-processing"
Oct 30 21:19:41.092910 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:41.092720    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="74cd503a-e270-489a-8876-6b27084a9981" containerName="payload-pre-processing"
Oct 30 21:19:41.097356 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:41.097337    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-545519-676cd8876c-bs7ks"
Oct 30 21:19:41.154260 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:41.154231    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-545519-676cd8876c-bs7ks"]
Oct 30 21:19:41.215863 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:41.215794    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-545519\""
Oct 30 21:19:41.216088 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:41.215960    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-545519-dockercfg-89bqw\""
Oct 30 21:19:41.255284 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:41.255244    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a66d5015-0765-4a22-ab8d-b3178cd831a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-545519-676cd8876c-bs7ks\" (UID: \"a66d5015-0765-4a22-ab8d-b3178cd831a2\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-545519-676cd8876c-bs7ks"
Oct 30 21:19:41.255459 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:41.255409    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdzjz\" (UniqueName: \"kubernetes.io/projected/a66d5015-0765-4a22-ab8d-b3178cd831a2-kube-api-access-sdzjz\") pod \"payload-pre-processing-e2e-authcfg-http-545519-676cd8876c-bs7ks\" (UID: \"a66d5015-0765-4a22-ab8d-b3178cd831a2\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-545519-676cd8876c-bs7ks"
Oct 30 21:19:41.356491 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:41.356417    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdzjz\" (UniqueName: \"kubernetes.io/projected/a66d5015-0765-4a22-ab8d-b3178cd831a2-kube-api-access-sdzjz\") pod \"payload-pre-processing-e2e-authcfg-http-545519-676cd8876c-bs7ks\" (UID: \"a66d5015-0765-4a22-ab8d-b3178cd831a2\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-545519-676cd8876c-bs7ks"
Oct 30 21:19:41.356491 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:41.356470    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a66d5015-0765-4a22-ab8d-b3178cd831a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-545519-676cd8876c-bs7ks\" (UID: \"a66d5015-0765-4a22-ab8d-b3178cd831a2\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-545519-676cd8876c-bs7ks"
Oct 30 21:19:41.357209 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:41.357187    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a66d5015-0765-4a22-ab8d-b3178cd831a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-545519-676cd8876c-bs7ks\" (UID: \"a66d5015-0765-4a22-ab8d-b3178cd831a2\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-545519-676cd8876c-bs7ks"
Oct 30 21:19:41.414787 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:41.414757    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdzjz\" (UniqueName: \"kubernetes.io/projected/a66d5015-0765-4a22-ab8d-b3178cd831a2-kube-api-access-sdzjz\") pod \"payload-pre-processing-e2e-authcfg-http-545519-676cd8876c-bs7ks\" (UID: \"a66d5015-0765-4a22-ab8d-b3178cd831a2\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-545519-676cd8876c-bs7ks"
Oct 30 21:19:41.710721 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:41.710635    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-545519-676cd8876c-bs7ks"
Oct 30 21:19:42.085311 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:42.085272    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-545519-676cd8876c-bs7ks"]
Oct 30 21:19:42.086402 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:19:42.086368    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda66d5015_0765_4a22_ab8d_b3178cd831a2.slice/crio-63d8d5c059c527a17aea135dd8a115808e909dea9e167a70ac0a84d2bd4d4d46 WatchSource:0}: Error finding container 63d8d5c059c527a17aea135dd8a115808e909dea9e167a70ac0a84d2bd4d4d46: Status 404 returned error can't find the container with id 63d8d5c059c527a17aea135dd8a115808e909dea9e167a70ac0a84d2bd4d4d46
Oct 30 21:19:42.953023 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:42.952986    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-545519-676cd8876c-bs7ks" event={"ID":"a66d5015-0765-4a22-ab8d-b3178cd831a2","Type":"ContainerStarted","Data":"5e7325ecbd3e4923706292ecd984d425f4232b2ee09246d0c8f2f793eeb599a4"}
Oct 30 21:19:42.953430 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:42.953045    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-545519-676cd8876c-bs7ks" event={"ID":"a66d5015-0765-4a22-ab8d-b3178cd831a2","Type":"ContainerStarted","Data":"63d8d5c059c527a17aea135dd8a115808e909dea9e167a70ac0a84d2bd4d4d46"}
Oct 30 21:19:42.953430 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:42.953212    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-545519-676cd8876c-bs7ks"
Oct 30 21:19:43.070113 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:43.070063    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-545519-676cd8876c-bs7ks" podStartSLOduration=2.070048459 podStartE2EDuration="2.070048459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:43.067862229 +0000 UTC m=+3194.984883769" watchObservedRunningTime="2026-10-30 21:19:43.070048459 +0000 UTC m=+3194.987069994"
Oct 30 21:19:43.752375 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:43.752336    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:43.976686 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:43.976639    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:46.514467 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:46.514428    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:46.731226 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:19:46.731185    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:19:48.882238 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:48.882197    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:51.950124 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:51.950083    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:19:53.883109 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:53.883070    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:53.961707 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:53.961680    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-545519-676cd8876c-bs7ks"
Oct 30 21:19:57.122276 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:57.122240    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:19:58.887714 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:19:58.887680    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:20:02.090988 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:20:02.090954    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a873e6d-jwgvb"]
Oct 30 21:20:02.140174 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:20:02.140142    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a873e6d-jwgvb"]
Oct 30 21:20:02.200634 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:20:02.200600    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa49bbb7-63da-418b-9643-f4869e67393b" path="/var/lib/kubelet/pods/aa49bbb7-63da-418b-9643-f4869e67393b/volumes"
Oct 30 21:20:02.988449 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:20:02.988415    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:20:04.276377 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:20:04.276344    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:20:11.200731 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:20:11.200691    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:20:11.360712 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:20:11.360673    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:20:14.426080 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:20:14.426041    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:20:16.726537 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:20:16.726499    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:20:19.370784 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:20:19.370748    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:20:24.368182 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:20:24.368143    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:20:29.263501 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:20:29.263464    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:20:34.191566 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:20:34.191515    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:20:39.173761 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:20:39.173727    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:20:44.172445 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:20:44.172403    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:20:44.330060 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:20:44.330025    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:20:46.727258 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:20:46.727220    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:20:46.848790 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:20:46.848758    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:20:49.081433 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:20:49.081396    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:20:54.118368 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:20:54.115227    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:20:57.462913 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:20:57.462873    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:20:58.969106 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:20:58.969064    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:21:03.963395 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:03.963356    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:21:08.571545 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:08.571461    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:21:08.883307 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:08.883214    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:21:12.689718 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:12.689680    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:21:13.914673 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:13.914635    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:21:16.726040 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:21:16.726006    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:21:18.801657 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:18.801623    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:21:24.004712 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:24.004678    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:21:28.911964 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:28.911934    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8qvb_91fe9e06-0c30-4000-bddb-2ba89cd048cc/console-operator/2.log"
Oct 30 21:21:28.919997 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:28.919968    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q4f4h_80cae79e-b60f-4efe-bb58-e77753a457e5/ovn-acl-logging/0.log"
Oct 30 21:21:28.929988 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:28.929968    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-l8qvb_91fe9e06-0c30-4000-bddb-2ba89cd048cc/console-operator/2.log"
Oct 30 21:21:28.937486 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:28.937468    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q4f4h_80cae79e-b60f-4efe-bb58-e77753a457e5/ovn-acl-logging/0.log"
Oct 30 21:21:29.181413 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:29.181321    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:21:33.740937 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:33.740903    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-545519-676cd8876c-bs7ks"]
Oct 30 21:21:34.019025 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:34.018932    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:21:34.225168 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:34.225127    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:21:34.557629 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:34.557590    5071 scope.go:122] "RemoveContainer" containerID="b9eee5b284d7a258af53f603261914fa93cd9b9f8a1789d54fdbb2fe0e6f14e9"
Oct 30 21:21:34.567992 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:34.567967    5071 scope.go:122] "RemoveContainer" containerID="984a03a50c6f359fc256a9abd9e464b81381961d75398b4f7791fbcdfcbfa725"
Oct 30 21:21:38.742212 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:38.742166    5071 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-545519-676cd8876c-bs7ks" podUID="a66d5015-0765-4a22-ab8d-b3178cd831a2" containerName="payload-pre-processing" containerID="cri-o://5e7325ecbd3e4923706292ecd984d425f4232b2ee09246d0c8f2f793eeb599a4" gracePeriod=25
Oct 30 21:21:38.992975 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:38.992916    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-545519-676cd8876c-bs7ks"
Oct 30 21:21:39.096600 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:39.096566    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a66d5015-0765-4a22-ab8d-b3178cd831a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a66d5015-0765-4a22-ab8d-b3178cd831a2-plugins-config-volume\") pod \"a66d5015-0765-4a22-ab8d-b3178cd831a2\" (UID: \"a66d5015-0765-4a22-ab8d-b3178cd831a2\") "
Oct 30 21:21:39.096762 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:39.096614    5071 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a66d5015-0765-4a22-ab8d-b3178cd831a2-kube-api-access-sdzjz\" (UniqueName: \"kubernetes.io/projected/a66d5015-0765-4a22-ab8d-b3178cd831a2-kube-api-access-sdzjz\") pod \"a66d5015-0765-4a22-ab8d-b3178cd831a2\" (UID: \"a66d5015-0765-4a22-ab8d-b3178cd831a2\") "
Oct 30 21:21:39.096958 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:39.096936    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a66d5015-0765-4a22-ab8d-b3178cd831a2-plugins-config-volume" pod "a66d5015-0765-4a22-ab8d-b3178cd831a2" (UID: "a66d5015-0765-4a22-ab8d-b3178cd831a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:21:39.098861 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:39.098832    5071 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a66d5015-0765-4a22-ab8d-b3178cd831a2-kube-api-access-sdzjz" pod "a66d5015-0765-4a22-ab8d-b3178cd831a2" (UID: "a66d5015-0765-4a22-ab8d-b3178cd831a2"). InnerVolumeSpecName "kube-api-access-sdzjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:21:39.198019 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:39.197982    5071 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a66d5015-0765-4a22-ab8d-b3178cd831a2-plugins-config-volume\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:21:39.198019 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:39.198012    5071 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdzjz\" (UniqueName: \"kubernetes.io/projected/a66d5015-0765-4a22-ab8d-b3178cd831a2-kube-api-access-sdzjz\") on node \"ip-10-0-139-149.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:21:39.575697 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:39.575664    5071 generic.go:356] "Generic (PLEG): container finished" podID="a66d5015-0765-4a22-ab8d-b3178cd831a2" containerID="5e7325ecbd3e4923706292ecd984d425f4232b2ee09246d0c8f2f793eeb599a4" exitCode=0
Oct 30 21:21:39.575907 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:39.575716    5071 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-545519-676cd8876c-bs7ks"
Oct 30 21:21:39.575907 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:39.575726    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-545519-676cd8876c-bs7ks" event={"ID":"a66d5015-0765-4a22-ab8d-b3178cd831a2","Type":"ContainerDied","Data":"5e7325ecbd3e4923706292ecd984d425f4232b2ee09246d0c8f2f793eeb599a4"}
Oct 30 21:21:39.575907 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:39.575768    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-545519-676cd8876c-bs7ks" event={"ID":"a66d5015-0765-4a22-ab8d-b3178cd831a2","Type":"ContainerDied","Data":"63d8d5c059c527a17aea135dd8a115808e909dea9e167a70ac0a84d2bd4d4d46"}
Oct 30 21:21:39.575907 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:39.575789    5071 scope.go:122] "RemoveContainer" containerID="5e7325ecbd3e4923706292ecd984d425f4232b2ee09246d0c8f2f793eeb599a4"
Oct 30 21:21:39.585292 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:39.585264    5071 scope.go:122] "RemoveContainer" containerID="5e7325ecbd3e4923706292ecd984d425f4232b2ee09246d0c8f2f793eeb599a4"
Oct 30 21:21:39.585569 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:21:39.585545    5071 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e7325ecbd3e4923706292ecd984d425f4232b2ee09246d0c8f2f793eeb599a4\": container with ID starting with 5e7325ecbd3e4923706292ecd984d425f4232b2ee09246d0c8f2f793eeb599a4 not found: ID does not exist" containerID="5e7325ecbd3e4923706292ecd984d425f4232b2ee09246d0c8f2f793eeb599a4"
Oct 30 21:21:39.585626 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:39.585580    5071 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e7325ecbd3e4923706292ecd984d425f4232b2ee09246d0c8f2f793eeb599a4"} err="failed to get container status \"5e7325ecbd3e4923706292ecd984d425f4232b2ee09246d0c8f2f793eeb599a4\": rpc error: code = NotFound desc = could not find container \"5e7325ecbd3e4923706292ecd984d425f4232b2ee09246d0c8f2f793eeb599a4\": container with ID starting with 5e7325ecbd3e4923706292ecd984d425f4232b2ee09246d0c8f2f793eeb599a4 not found: ID does not exist"
Oct 30 21:21:39.742273 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:39.742242    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-545519-676cd8876c-bs7ks"]
Oct 30 21:21:39.791928 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:39.791893    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-545519-676cd8876c-bs7ks"]
Oct 30 21:21:40.201156 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:40.201126    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a66d5015-0765-4a22-ab8d-b3178cd831a2" path="/var/lib/kubelet/pods/a66d5015-0765-4a22-ab8d-b3178cd831a2/volumes"
Oct 30 21:21:43.306428 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:43.306392    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:21:44.602468 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:44.602425    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:21:46.319057 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:46.319017    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:21:46.727033 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:21:46.726951    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:21:47.225188 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:47.225150    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:21:47.780994 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:47.780959    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:21:52.395112 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:52.395073    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:21:52.590499 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:52.590452    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:21:52.889664 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:52.889607    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:21:57.859687 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:21:57.859650    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:22:01.051160 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:22:01.051127    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z7hvt"]
Oct 30 21:22:01.100878 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:22:01.100843    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z7hvt"]
Oct 30 21:22:02.200947 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:22:02.200910    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05076e73-051d-479a-990e-93bf067ede16" path="/var/lib/kubelet/pods/05076e73-051d-479a-990e-93bf067ede16/volumes"
Oct 30 21:22:02.638389 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:22:02.638353    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:22:10.469432 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:22:10.469382    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:22:10.689734 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:22:10.689697    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:22:15.170379 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:22:15.170338    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:22:16.847848 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:22:16.847079    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:22:36.153720 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:22:36.153681    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:22:38.053816 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:22:38.053738    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t7zm7"]
Oct 30 21:22:38.103301 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:22:38.103270    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t7zm7"]
Oct 30 21:22:38.201095 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:22:38.201065    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7687aa1f-92bb-4fca-8a33-515fa3a2241f" path="/var/lib/kubelet/pods/7687aa1f-92bb-4fca-8a33-515fa3a2241f/volumes"
Oct 30 21:22:44.998297 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:22:44.998265    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:22:46.726791 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:22:46.726747    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:22:47.530710 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:22:47.530678    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:22:58.199340 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:22:58.199302    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:23:08.691781 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:08.691743    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-pwbqs"]
Oct 30 21:23:09.056976 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:09.056940    5071 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6bee289-2qbmm"]
Oct 30 21:23:09.109237 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:09.109199    5071 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6bee289-2qbmm"]
Oct 30 21:23:10.239847 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:10.236257    5071 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e673006-6fdf-4ac9-8186-afe96fe4f5df" path="/var/lib/kubelet/pods/7e673006-6fdf-4ac9-8186-afe96fe4f5df/volumes"
Oct 30 21:23:16.727067 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:23:16.727028    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:23:20.211748 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:20.211714    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-lkmv2_2064d6a5-f7fb-46fa-b655-7366285e6148/manager/0.log"
Oct 30 21:23:20.978113 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:20.978021    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6565584cfb-zrltv_fe81a376-1de8-4c13-8d8a-49c638d37d00/manager/0.log"
Oct 30 21:23:22.755352 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:22.755322    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6d9876d865-9rcvl_d626d86a-5d08-4929-af30-3d124bce8f7d/maas-api/0.log"
Oct 30 21:23:26.019791 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:26.019755    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-4j4v2_da53ef69-95be-48e9-971e-4168bf928443/manager/0.log"
Oct 30 21:23:26.243846 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:26.243782    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-tvzgz_e9bd0a5f-c80e-416d-a40c-6fd2def09183/manager/0.log"
Oct 30 21:23:26.759879 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:26.759844    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-lwcph_ab471889-9dee-461d-a125-7cd405093e0a/manager/0.log"
Oct 30 21:23:27.148173 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:27.148148    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-pwbqs_8ea98f67-8b88-46cb-abac-ed7153834e6a/limitador/0.log"
Oct 30 21:23:28.337931 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:28.337898    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2_97751fc9-3d1c-451d-b2e3-7fbc522d3ad9/istio-proxy/0.log"
Oct 30 21:23:28.587037 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:28.587007    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-jscm6_21b86bc7-8a12-4fdc-9074-b044bbf04f4c/discovery/0.log"
Oct 30 21:23:29.835086 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:29.835058    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-cv4kg_9be8fbd1-6cb8-4718-8cfe-a45894b5d33e/istio-proxy/0.log"
Oct 30 21:23:30.607037 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:30.607004    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-pmwpk_a367ee40-c5ea-44f3-92c6-d1b5cf1141b3/istio-proxy/0.log"
Oct 30 21:23:30.893630 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:30.893559    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-s6hth_ecd40162-84fb-4838-ae3b-bafa49f9cdb9/istio-proxy/0.log"
Oct 30 21:23:31.612476 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:31.612440    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-xhg2f_562c4391-03f9-470d-a007-b4931bd3f421/istio-proxy/0.log"
Oct 30 21:23:31.845190 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:31.845150    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj_223f5d8a-238d-49f4-9951-0a6a2019cee7/istio-proxy/0.log"
Oct 30 21:23:32.508430 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:32.508392    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-666dd86ff-dnnq5_533fb89e-0b69-4c79-9cbf-7baa5f725219/router/0.log"
Oct 30 21:23:34.258601 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:34.258569    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds_5b24c7ce-f038-48b4-bc61-e1b6040b8e83/main/0.log"
Oct 30 21:23:34.606808 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:34.606776    5071 scope.go:122] "RemoveContainer" containerID="cb398cb9a6bd62ae58d22bae7ae02fbee58f57c8376397898d615ab9ffbade50"
Oct 30 21:23:34.615921 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:34.615897    5071 scope.go:122] "RemoveContainer" containerID="725fe8f0e08b86f2694083d9030500c890394088e8169a2f8b3b18a6033ed6a9"
Oct 30 21:23:45.261318 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:45.261280    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:23:46.289084 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:46.289019    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-lkmv2_2064d6a5-f7fb-46fa-b655-7366285e6148/manager/0.log"
Oct 30 21:23:46.727255 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:23:46.727162    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:23:47.070652 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:47.070621    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6565584cfb-zrltv_fe81a376-1de8-4c13-8d8a-49c638d37d00/manager/0.log"
Oct 30 21:23:47.886977 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:47.886928    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:23:48.692297 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:48.692259    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6d9876d865-9rcvl_d626d86a-5d08-4929-af30-3d124bce8f7d/maas-api/0.log"
Oct 30 21:23:51.837009 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:51.836975    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-4j4v2_da53ef69-95be-48e9-971e-4168bf928443/manager/0.log"
Oct 30 21:23:52.099134 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:52.099045    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-tvzgz_e9bd0a5f-c80e-416d-a40c-6fd2def09183/manager/0.log"
Oct 30 21:23:52.564838 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:52.564791    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-lwcph_ab471889-9dee-461d-a125-7cd405093e0a/manager/0.log"
Oct 30 21:23:52.956394 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:52.956307    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-pwbqs_8ea98f67-8b88-46cb-abac-ed7153834e6a/limitador/0.log"
Oct 30 21:23:54.347842 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:54.347793    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2_97751fc9-3d1c-451d-b2e3-7fbc522d3ad9/istio-proxy/0.log"
Oct 30 21:23:54.707038 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:54.706958    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-jscm6_21b86bc7-8a12-4fdc-9074-b044bbf04f4c/discovery/0.log"
Oct 30 21:23:56.425583 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:56.425548    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-cv4kg_9be8fbd1-6cb8-4718-8cfe-a45894b5d33e/istio-proxy/0.log"
Oct 30 21:23:57.340598 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:57.340565    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-pmwpk_a367ee40-c5ea-44f3-92c6-d1b5cf1141b3/istio-proxy/0.log"
Oct 30 21:23:57.651524 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:57.651448    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-s6hth_ecd40162-84fb-4838-ae3b-bafa49f9cdb9/istio-proxy/0.log"
Oct 30 21:23:58.451658 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:58.451623    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-xhg2f_562c4391-03f9-470d-a007-b4931bd3f421/istio-proxy/0.log"
Oct 30 21:23:58.573517 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:58.573479    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:23:58.699064 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:58.699032    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-856b6dffcf-jm7fj_223f5d8a-238d-49f4-9951-0a6a2019cee7/istio-proxy/0.log"
Oct 30 21:23:59.407209 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:23:59.407166    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-666dd86ff-dnnq5_533fb89e-0b69-4c79-9cbf-7baa5f725219/router/0.log"
Oct 30 21:24:00.948987 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:24:00.948955    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fdkds_5b24c7ce-f038-48b4-bc61-e1b6040b8e83/main/0.log"
Oct 30 21:24:16.727671 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:24:16.727634    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:24:20.242110 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:24:20.242069    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:24:20.448003 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:24:20.447960    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:24:30.508242 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:24:30.508207    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:24:31.667427 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:24:31.667389    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6565584cfb-zrltv_fe81a376-1de8-4c13-8d8a-49c638d37d00/manager/0.log"
Oct 30 21:24:32.241150 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:24:32.241116    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6565584cfb-zrltv_fe81a376-1de8-4c13-8d8a-49c638d37d00/manager/0.log"
Oct 30 21:24:34.639774 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:24:34.639742    5071 scope.go:122] "RemoveContainer" containerID="fee0d9c3a15fa10aa2cf160e4ff97c2605e5a34af6155d0cc2765a29e8edfcd7"
Oct 30 21:24:45.582527 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:24:45.582487    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:24:46.727184 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:24:46.727146    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:24:48.206397 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:24:48.206355    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:24:58.867048 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:24:58.867011    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:25:12.781415 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:12.781365    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6565584cfb-zrltv_fe81a376-1de8-4c13-8d8a-49c638d37d00/manager/0.log"
Oct 30 21:25:13.365862 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:13.365816    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6565584cfb-zrltv_fe81a376-1de8-4c13-8d8a-49c638d37d00/manager/0.log"
Oct 30 21:25:14.731325 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:14.731285    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:25:14.901169 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:14.901129    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:25:16.727670 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:25:16.727636    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:25:23.986365 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:23.986327    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bqh9l_88d9611e-c7ed-490b-b401-530d1649c873/global-pull-secret-syncer/0.log"
Oct 30 21:25:25.763838 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:25.763793    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-brf75_08b2f787-ad25-47f9-8ef8-724308458f88/connection-checker/0.log"
Oct 30 21:25:26.190289 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:26.190259    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-c5b2k_78c7e317-9d58-4752-9804-92ac45b5afb7/connection-checker/0.log"
Oct 30 21:25:26.618468 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:26.618440    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-kgsjn_f4913de8-6db5-413a-938f-abc50a5460e7/connection-checker/0.log"
Oct 30 21:25:27.090829 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:27.090798    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log"
Oct 30 21:25:28.817336 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:28.817302    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-149.us-west-1.compute.internal_9d9f66ca5c88277c8c76e3090f63988d/haproxy/0.log"
Oct 30 21:25:33.444516 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:33.444477    5071 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-x4r57/perf-node-gather-daemonset-qjqhg"]
Oct 30 21:25:33.444970 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:33.444954    5071 cpu_manager.go:414] "containerMap: removing container" podUID="a66d5015-0765-4a22-ab8d-b3178cd831a2" containerName="payload-pre-processing"
Oct 30 21:25:33.445018 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:33.444971    5071 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a66d5015-0765-4a22-ab8d-b3178cd831a2" containerName="payload-pre-processing"
Oct 30 21:25:33.445088 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:33.445078    5071 memory_manager.go:365] "RemoveStaleState removing state" podUID="a66d5015-0765-4a22-ab8d-b3178cd831a2" containerName="payload-pre-processing"
Oct 30 21:25:33.448137 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:33.448122    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-qjqhg"
Oct 30 21:25:33.501914 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:33.501890    5071 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-x4r57\"/\"default-dockercfg-sq655\""
Oct 30 21:25:33.520380 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:33.520339    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhz89\" (UniqueName: \"kubernetes.io/projected/6e605fc6-6e4b-456a-9743-7cfb19fd8b20-kube-api-access-xhz89\") pod \"perf-node-gather-daemonset-qjqhg\" (UID: \"6e605fc6-6e4b-456a-9743-7cfb19fd8b20\") " pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-qjqhg"
Oct 30 21:25:33.520380 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:33.520378    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6e605fc6-6e4b-456a-9743-7cfb19fd8b20-proc\") pod \"perf-node-gather-daemonset-qjqhg\" (UID: \"6e605fc6-6e4b-456a-9743-7cfb19fd8b20\") " pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-qjqhg"
Oct 30 21:25:33.520628 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:33.520403    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6e605fc6-6e4b-456a-9743-7cfb19fd8b20-lib-modules\") pod \"perf-node-gather-daemonset-qjqhg\" (UID: \"6e605fc6-6e4b-456a-9743-7cfb19fd8b20\") " pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-qjqhg"
Oct 30 21:25:33.520628 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:33.520579    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6e605fc6-6e4b-456a-9743-7cfb19fd8b20-podres\") pod \"perf-node-gather-daemonset-qjqhg\" (UID: \"6e605fc6-6e4b-456a-9743-7cfb19fd8b20\") " pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-qjqhg"
Oct 30 21:25:33.520628 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:33.520626    5071 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e605fc6-6e4b-456a-9743-7cfb19fd8b20-sys\") pod \"perf-node-gather-daemonset-qjqhg\" (UID: \"6e605fc6-6e4b-456a-9743-7cfb19fd8b20\") " pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-qjqhg"
Oct 30 21:25:33.547451 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:33.547420    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-4j4v2_da53ef69-95be-48e9-971e-4168bf928443/manager/0.log"
Oct 30 21:25:33.551059 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:33.551035    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x4r57\"/\"kube-root-ca.crt\""
Oct 30 21:25:33.551213 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:33.551131    5071 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x4r57\"/\"openshift-service-ca.crt\""
Oct 30 21:25:33.553236 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:33.553206    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x4r57/perf-node-gather-daemonset-qjqhg"]
Oct 30 21:25:33.621255 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:33.621229    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhz89\" (UniqueName: \"kubernetes.io/projected/6e605fc6-6e4b-456a-9743-7cfb19fd8b20-kube-api-access-xhz89\") pod \"perf-node-gather-daemonset-qjqhg\" (UID: \"6e605fc6-6e4b-456a-9743-7cfb19fd8b20\") " pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-qjqhg"
Oct 30 21:25:33.621398 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:33.621260    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6e605fc6-6e4b-456a-9743-7cfb19fd8b20-proc\") pod \"perf-node-gather-daemonset-qjqhg\" (UID: \"6e605fc6-6e4b-456a-9743-7cfb19fd8b20\") " pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-qjqhg"
Oct 30 21:25:33.621398 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:33.621282    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6e605fc6-6e4b-456a-9743-7cfb19fd8b20-lib-modules\") pod \"perf-node-gather-daemonset-qjqhg\" (UID: \"6e605fc6-6e4b-456a-9743-7cfb19fd8b20\") " pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-qjqhg"
Oct 30 21:25:33.621398 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:33.621346    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6e605fc6-6e4b-456a-9743-7cfb19fd8b20-podres\") pod \"perf-node-gather-daemonset-qjqhg\" (UID: \"6e605fc6-6e4b-456a-9743-7cfb19fd8b20\") " pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-qjqhg"
Oct 30 21:25:33.621398 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:33.621365    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6e605fc6-6e4b-456a-9743-7cfb19fd8b20-proc\") pod \"perf-node-gather-daemonset-qjqhg\" (UID: \"6e605fc6-6e4b-456a-9743-7cfb19fd8b20\") " pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-qjqhg"
Oct 30 21:25:33.621579 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:33.621423    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6e605fc6-6e4b-456a-9743-7cfb19fd8b20-podres\") pod \"perf-node-gather-daemonset-qjqhg\" (UID: \"6e605fc6-6e4b-456a-9743-7cfb19fd8b20\") " pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-qjqhg"
Oct 30 21:25:33.621579 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:33.621434    5071 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e605fc6-6e4b-456a-9743-7cfb19fd8b20-sys\") pod \"perf-node-gather-daemonset-qjqhg\" (UID: \"6e605fc6-6e4b-456a-9743-7cfb19fd8b20\") " pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-qjqhg"
Oct 30 21:25:33.621579 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:33.621452    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6e605fc6-6e4b-456a-9743-7cfb19fd8b20-lib-modules\") pod \"perf-node-gather-daemonset-qjqhg\" (UID: \"6e605fc6-6e4b-456a-9743-7cfb19fd8b20\") " pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-qjqhg"
Oct 30 21:25:33.621579 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:33.621505    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e605fc6-6e4b-456a-9743-7cfb19fd8b20-sys\") pod \"perf-node-gather-daemonset-qjqhg\" (UID: \"6e605fc6-6e4b-456a-9743-7cfb19fd8b20\") " pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-qjqhg"
Oct 30 21:25:33.679892 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:33.679863    5071 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhz89\" (UniqueName: \"kubernetes.io/projected/6e605fc6-6e4b-456a-9743-7cfb19fd8b20-kube-api-access-xhz89\") pod \"perf-node-gather-daemonset-qjqhg\" (UID: \"6e605fc6-6e4b-456a-9743-7cfb19fd8b20\") " pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-qjqhg"
Oct 30 21:25:33.760556 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:33.760488    5071 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-qjqhg"
Oct 30 21:25:33.969324 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:33.969284    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-tvzgz_e9bd0a5f-c80e-416d-a40c-6fd2def09183/manager/0.log"
Oct 30 21:25:34.135955 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:34.135929    5071 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x4r57/perf-node-gather-daemonset-qjqhg"]
Oct 30 21:25:34.138894 ip-10-0-139-149 kubenswrapper[5071]: W1030 21:25:34.138864    5071 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6e605fc6_6e4b_456a_9743_7cfb19fd8b20.slice/crio-1bc554e0e14ca368fa69730b487a8e7e0615fb651c8d1c6cc10ff819fb570414 WatchSource:0}: Error finding container 1bc554e0e14ca368fa69730b487a8e7e0615fb651c8d1c6cc10ff819fb570414: Status 404 returned error can't find the container with id 1bc554e0e14ca368fa69730b487a8e7e0615fb651c8d1c6cc10ff819fb570414
Oct 30 21:25:34.901955 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:34.901926    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-lwcph_ab471889-9dee-461d-a125-7cd405093e0a/manager/0.log"
Oct 30 21:25:35.000876 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:35.000833    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-qjqhg" event={"ID":"6e605fc6-6e4b-456a-9743-7cfb19fd8b20","Type":"ContainerStarted","Data":"0dea818b55b7642ee58cf17e6d70e314862f242fd672b87b67e67106f88ec428"}
Oct 30 21:25:35.001051 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:35.000884    5071 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-qjqhg" event={"ID":"6e605fc6-6e4b-456a-9743-7cfb19fd8b20","Type":"ContainerStarted","Data":"1bc554e0e14ca368fa69730b487a8e7e0615fb651c8d1c6cc10ff819fb570414"}
Oct 30 21:25:35.001051 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:35.000978    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-qjqhg"
Oct 30 21:25:35.318068 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:35.318044    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-pwbqs_8ea98f67-8b88-46cb-abac-ed7153834e6a/limitador/0.log"
Oct 30 21:25:41.017898 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:41.017804    5071 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-qjqhg"
Oct 30 21:25:41.135901 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:41.135854    5071 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-x4r57/perf-node-gather-daemonset-qjqhg" podStartSLOduration=8.135838553 podStartE2EDuration="8.135838553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:25:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:35.115401461 +0000 UTC m=+3547.032422991" watchObservedRunningTime="2026-10-30 21:25:41.135838553 +0000 UTC m=+3553.052860082"
Oct 30 21:25:41.520213 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:41.520180    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-9rs22_d5312b70-82df-4e75-a300-b8e1d63a75f3/cluster-monitoring-operator/0.log"
Oct 30 21:25:45.385485 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:45.385449    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mvkzt_e21d7e79-1c77-4815-b43d-603881b3252c/node-exporter/0.log"
Oct 30 21:25:45.800609 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:45.800576    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mvkzt_e21d7e79-1c77-4815-b43d-603881b3252c/kube-rbac-proxy/0.log"
Oct 30 21:25:45.933951 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:45.933917    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:25:46.222915 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:46.222838    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mvkzt_e21d7e79-1c77-4815-b43d-603881b3252c/init-textfile/0.log"
Oct 30 21:25:46.727969 ip-10-0-139-149 kubenswrapper[5071]: E1030 21:25:46.727923    5071 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tk6nb" containerName="extractor"
Oct 30 21:25:48.546814 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:48.546775    5071 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4khkm_89ed3d9f-e057-40a3-add4-d6374380be1f/konnectivity-agent/0.log" limit=1024
Oct 30 21:25:56.883631 ip-10-0-139-149 kubenswrapper[5071]: I1030 21:25:56.883599    5071 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-59cf4dc6bfhzsv2_97751fc9-3d1c-451d-b2e3-7fbc522d3ad9/istio-proxy/0.log"